search

msmbuilder / osprey

🦅Hyperparameter optimization for machine learning pipelines 🦅
http://msmbuilder.org/osprey
Apache License 2.0
74 stars 26 forks source link

Skip re-running a trial with identical params #215

Closed synapticarbors closed 7 years ago

synapticarbors commented 7 years ago

This implements a change to the worker so it does not repeat trials on identical parameter sets as suggested in https://github.com/msmbuilder/osprey/issues/149. I've added a parameter to the config max_param_suggestion_retries which sets a limit on how many times strategy.suggest is called when it produces a new param set that already is in the history. The default is to create a trial ignoring previous history.

This seems particularly helpful when restarting a run that uses a grid search so you don't re-calculate for already visited points on the grid. It may, however be better to do the check internal to the grid search's suggest method to "fast-forward" to the first point that hasn't already been visited. The current method only runs unique param combinations even when the search space has added or removed hyperparameters that aren't visible to suggest.

This was my first stab at this and I didn't consult with the Osprey developers first, so I'm not sure if it is the preferred strategy for doing this. I'd appreciate any feedback/code review and I'd be happy to make any changes necessary to get it merged.

synapticarbors commented 7 years ago

@cxhernandez I updated the changelog and added a section to the config docs.

cxhernandez commented 7 years ago

Awesome! Thanks so much!

synapticarbors commented 7 years ago

Thanks @cxhernandez. A few follow-up questions:

cxhernandez commented 7 years ago

When do you expect v1.2 is going to be released?

I'm thinking it'll happen alongside the next release of MSMBuilder, which should happen within the next week.

Do you have any interest in me putting together a recipe for Osprey on conda-forge? I can probably do hyperopt as well. I know that MSMBuilder is on conda-forge now as well as mdtraj.

That would be great!