Add cross-val diagnostic figure

MPoL-dev / MPoL

A flexible Python platform for Regularized Maximum Likelihood imaging

https://mpol-dev.github.io/MPoL/

MIT License

33 stars 11 forks source link

Add cross-val diagnostic figure #214

Closed jeffjennings closed 9 months ago

jeffjennings commented 9 months ago

NOTE: This should only be reviewed after https://github.com/MPoL-dev/MPoL/pull/213 is merged into main and main into here.

Adds the plotting function plot.crossval_diagnostics_fig to produce a diagnostic figure using a CrossValidate object. Figure shows a loss evolution and CV score for each k-fold.
Sets train.TrainTest.trainloss_lambda_guess to only run if 1+ of the regularizers used has guess of True (user-provided).

jeffjennings commented 9 months ago

Example figure: IMLup_crossval_diagnostics

jeffjennings commented 9 months ago

Yes, I should have the pipeline script done in the next few days. And yeah it's true that consistently, even with reasonably large dartboard cells, the loss profiles remain pretty similar. I could test how much of a difference it makes to increase the dartboard cell size or decrease the number of k-folds, or other ways to distinguish better between cross-vals with different regularizers/strengths. I don't know, I guess if many regularizer/strength combinations fit the observed data very similarly, maybe the model just needs more data to be able to distinguish between these (i.e., increase the predictive accuracy of unobserved points)...I'm going to make this its own issue. thanks for the review!

update: made this an issue, #215