PathologyDataScience / glimr

A simplified wrapper for hyperparameter search with Ray Tune.
Apache License 2.0
1 stars 0 forks source link

Model averaging of top trials #28

Open cooperlab opened 1 year ago

cooperlab commented 1 year ago

We could average top trials or compose an ensemble of sensitive and specific models for prediction. This should improve accuracy and also enables calculation of uncertainties. Perhaps this cannot be generalized for all applications and should be handled in the application libraries instead.

cooperlab commented 1 year ago

@lawrence-chillrud one idea is to accept a list of checkpoint directories as inputs and build and save a new model that returns a simple average. This could be similar to the top_k function.

If you are thinking more about a Bayesian or trainable approach to averaging that is conditional on inputs we would need to define what that is.

cooperlab commented 1 year ago

Given a validation set, we could identify models that have less correlated outputs for inclusion in the ensemble.