search

microsoft / MLOS

MLOS is a project to enable autotuning for systems.
https://microsoft.github.io/MLOS
MIT License
136 stars 64 forks source link

Latest version of FLAML changes optimization behavior #839

Open bpkroth opened 1 month ago

bpkroth commented 1 month ago

Recently a new version (v2.2.0) of FLAML was released and it introduces some new warnings and behavior:

INFO     flaml.tune.searcher.blendsearch:blendsearch.py:169 No low-cost partial config given to the search algorithm. For cost-frugal search, consider providing low-cost values for cost-related hps via 'low_cost_partial_config'. More info can be found at https://microsoft.github.io/FLAML/docs/FAQ#about-low_cost_partial_config-in-tune
WARNING  flaml.tune.searcher.suggestion:suggestion.py:555 You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.
WARNING  flaml.tune.searcher.suggestion:suggestion.py:555 You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `config` argument to `tune.run()` instead.
FAILED mlos_bench/mlos_bench/tests/optimizers/toy_optimization_loop_test.py::test_flaml_optimization_loop - assert 63.77786073141286 == 60.15 ± 6.0e-01

  comparison failed
  Obtained: 63.77786073141286
  Expected: 60.15 ± 6.0e-01

We should investigate and fix these.

For now, I'm creating a PR to simply pin the version to avoid them until we have time to do that.