search

microsoft / FLAML

A fast library for AutoML and tuning. Join our Discord: https://discord.gg/Cppx2vSPVP.
https://microsoft.github.io/FLAML/
MIT License
3.93k stars 513 forks source link

How can I gracefully continue an optimization run once completed? #514

Closed EgorKraevTransferwise closed 2 years ago

EgorKraevTransferwise commented 2 years ago

I would like to first do a tune.run() over a certain subset of the search space, with a certain time budget, then depending on the results continue that fit while possibly using a different subset of the same search space, but taking into account the previous runs.

What is the most graceful way of doing that? I could take the previous results from .results and transform them them into points_to_evaluate and evaluated _rewards for the next run, but this seems rather clumsy - is there not a way to tell tune during a second fit() call to take into account the previous .results?

qingyun-wu commented 2 years ago

Hi @EgorKraevTransferwise,

What you said, "take the previous results from .results and transform them into points_to_evaluate and evaluated _rewards for the next run," is the right way to do it currently. By clumsy, do you mean it requires additional code? Or something else? Presumably, it should just take 1 or 2 additional lines of code. let me know if you need help.

EgorKraevTransferwise commented 2 years ago

Thanks! By 'clumsy', I mean for example that this approach doesn't allow to do a run, then both resume the run so taking into account the earlier attempts, and at the same time prescribe some new config points for it to try for which the score is as yet unknown (useful, for example, when the resuming run wants to add a couple of new model types to the search space). Also, if I do it this way, will the pre-supplied values be part of the 'results' property after the run is finished? If not, that would make it somewhat fiddly to resume more than once.

qingyun-wu commented 2 years ago

Hi @EgorKraevTransferwise, thank you for your explanation. We plan to improve the way to accept points_to_evaluate and evaluated _rewards such that tune.run can both accept the evaluated points and new points. This should be able to resolve the first clumsy issue you mentioned. I will keep you posted about this improvement.

For the second question, currently, the pre-supplied values are not part of the result property after the run is finished. Adding a function to merge the pre-supplied values and the newly generated results should be easy. But how to add it in flaml in fact depends on your needs. I wonder whether you could help create a PR to add such a function depending on your needs. We are willing to help on the PR if needed.

Thank you, and let me know what you think about the suggested plans.

qingyun-wu commented 2 years ago

Hi @EgorKraevTransferwise,

The two needs you mentioned should have been resolved by the above PR, which was released via v1.0.1 yesterday. (1) revised points_to_evaluate and evaluated_reward, which allows you to append new points that do not have reward at the end of points_to_evaluate (2) revised results property which includes the initial results provided via evaluated_reward Can you please test them to see how they work? Let me know if they do not work well for you.

Thank you!

TimoFlesch commented 2 years ago

Hi, I'm working with Egor on this. Thank you very much for the quick replies! I've just implemented the warm start by supplying evaluated_rewards and points_to_evaluate which includes all configs from the result dict of the previous run, as well as some additional configs which we hadn't tried previously. The tuner proceeds with those, as intended. However, once that second fit has completed, the returned results dict only contains the trials and results from that second run. Am I missing something? Thanks again for your support!

qingyun-wu commented 2 years ago

Hi @TimoFlesch, thank you for following up. The results are in fact added to the results property of the searcher instead of the returned ExperimentAnlaysis class. Sorry about the confusion. Please find an example in this link: https://github.com/microsoft/FLAML/blob/main/test/tune/test_tune.py#L246

TimoFlesch commented 2 years ago

Hi @qingyun-wu Ok that makes perfect sense, thanks for the quick reply! I can confirm that it works as intended, this issue can be closed now.

qingyun-wu commented 2 years ago

Hi @qingyun-wu Ok that makes perfect sense, thanks for the quick reply! I can confirm that it works as intended, this issue can be closed now.

Hi @TimoFlesch, and @EgorKraevTransferwise,

Sure. You are welcome. If you haven't done so, can you please do a quick survey about FLAML: https://docs.google.com/forms/d/e/1FAIpQLSd0u4BbSzr-m9hgM8rmPlYB_JJoFXeHhuoXI9YFbHDRWrDlOQ/viewform?usp=sf_link

Thank you!