search

openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
399 stars 132 forks source link

Add `flaml_benchmark` #528

Closed PGijsbers closed 1 year ago

PGijsbers commented 1 year ago

Supersedes #414

todo: add a check that flaml can't be invoked if flaml_benchmark is already installed (it is currently not possible to switch behavior just based on hyperparameter configuration).

PGijsbers commented 1 year ago

@LittleLittleCloud We would like to support the benchmark mode of flaml, which you started with in #414. I had a closer look, and since psutil is installed by default in the environment (an amlb dependency), and xgboost is a standard dependency of flaml, it seems that catboost is the only difference (benchmark vs vanilla install with amlb dependencies). So I resolved to emulating "benchmark" mode through this work around: install catboost when running flaml in benchmark mode, and uninstall catboost otherwise. I am not installing benchmark directly because of https://github.com/microsoft/FLAML/issues/1060

Does this seem correct? I would still want to change this in the future, an easier way to switch between different modes would be appreciated.