KFold during Scan to minimize split effect

[beeb]

When running Scan to perform a search in the parameters space, there is no option to assess accuracy or other metrics using K-Fold cross-validation.

As such, when the model is sensitive to the dataset split (one reason could be there are few samples), the calculated accuracy used to select the best model has little meaning.

Each run provides a vastly different result and no ideal candidate can be identified.

Would it be possible to add cross-validation using KFold directly when running Scan in order to have a solid accuracy metric for the investigated candidates?

I modified the workflow a tad to run each candidate more than once (with a fraction_limit > 1) and processing the data to compute the mean of each metric from the same parameters set, but they all use the same data split so it's not cross-validation.

One could imagine running Scan first to generate all candidate models, then performing Evaluate with KFold cross-validation on each of the candidates, maybe?

Thanks!

EDIT: just came accross evaluate_models from the Scan object.. Could be that it does what I need, even if it means running it on all models.

[mikkokotila]

That's right, check out the section for evaluate_models in the docs.

The wider topic is of great interest as well. I spent sometime seeing if kfold could be directly implemented as a metric in Keras, but I think it's not logical because of the way a Keras metric works. That would been ideal.

The other topic you mention is seeing how different validation splits effect. I think that could be a meaningful variable in the experiment together with hyperparameters. Would have to think about it though as it is one level higher in terms of abstraction; we are then abstracting the Scan() configuration as opposed to Keras model configuration. This could become part of AutoScan ... check it out if had not already.

[beeb]

Hey thanks for the reply! I'm in the process of upgrading to v0.5 exactly because I saw AutoScan in the docs and it looks amazing. For now I think I'm happy with using evaluate_models, but I have to finish the upgrade to v0.5 (checking if everything works) because that's when you added the option to define the ml_task. And I'm doing mostly regression these days.

Best, Val

[mikkokotila]

I suggest moving directly to v.0.6.2 which is the current master.

[beeb]

Isn't v0.5 the LTS version? Also all my dependencies are managed via pip so it's easier to stay on a version that's available on pypi.

[mikkokotila]

442 contains instructions on how to achieve this currently in Talos.

Closing as resolved.