convergence warning for building kl_surrogate_model

[FariborzDaneshvar-NOAA]

The surrogate_from_training_set() function (in uncertainty_quantification/surrogate.py) uses sklearn.linear_model.LinearRegression() to fit a surrogate model on the training dataset. But I noticed that it is not converging!:

/.../lib/python3.10/site-packages/sklearn/linear_model/_coordinate_descent.py:678: 
ConvergenceWarning: Objective did not converge.
You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. 
Duality gap: 2.635e-03, tolerance: 1.936e-03
  model = cd_fast.enet_coordinate_descent(
...

I tested it on different training datasets (with 23 and 30 members), but still getting the same message.

@SorooshMani-NOAA @WPringle I assume increasing the size of the training datasets is not a favorable option. Would you recommend increasing max_iterations (default is1000) or decreasing the tolerance (default is 0.0001), or both? Any suggestion?

For my reference: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNetCV.html

[FariborzDaneshvar-NOAA]

I didn't get warnings for tol=0.001

[SorooshMani-NOAA]

I think we should see how increasing the number of iterations would impact the final results compared to increasing the tolerance (which just gets rid of the warnings)

[WPringle]

I have seen these warnings before but never found the result to be strange, but it could be worth checking to see if any impact on the final results by increasing iterations as Soroosh said.

[FariborzDaneshvar-NOAA]

@SorooshMani-NOAA @WPringle thanks for your suggestions. I did a quick test (see below), and in summary, it looks like the differences are negligible, and it's safe to say that the current setup is working fine. I tested three scenarios:

1. Default (tol=0.0001, max_iter=1000) (Returned ConvergenceWarning)
2. More iterations (tol=0.0001, max_iter=2000) (Converged)
3. Lower tolerance (tol=0.001, max_iter=1000) (Converged)

Here are KL surrogates for six modes of these three scenarios, proving that the performance of these three scenarios are pretty much the same:

Default	More iterations	Lower tolerance

RMSE	modes / scenario	Default	More iterations	Lower tolerance
mode 1	0.026	0.026	0.024
mode 2	0.016	0.016	0.017
mode 3	0.161	0.161	0.155
mode 4	0.023	0.023	0.007
mode 5	0.975	0.975	0.975
mode 6	0.903	0.903	0.903

CORR	modes / scenario	Default	More iterations	Lower tolerance
mode 1	1.0	1.0	1.0
mode 2	1.0	1.0	1.0
mode 3	0.989	0.989	0.99
mode 4	1.0	1.0	1.0
mode 5	0.0	0.0	0.0
mode 6	0.438	0.438	0.438