search

automl / HPOBench

Collection of hyperparameter optimization benchmark problems
Apache License 2.0
139 stars 36 forks source link

Where to find the best-known values of all benchmarks? #158

Closed tyf0416 closed 2 years ago

tyf0416 commented 2 years ago

I am trying to study the benchmark library and test some customized optimizers according to the same framework you have published. But I am wondering where could I find the best-known values of all benchmarks applied in the paper "HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO"? Are they accessible in the experimental code, or somewhere? Hope for your message.

Neeratyoy commented 2 years ago

Hi, The best-known value or the optimum is only available for the TabularBenchmarks.

KEggensperger commented 2 years ago

And for the remaining benchmarks, these values are in the experiment code and can be found here (with some comments on how we obtained them).

PhMueller commented 2 years ago

Maybe it is a good idea to store the best available values (if available) in the benchmark or somewhere in the readme? @KEggensperger

The latter would be easier for now.

KEggensperger commented 2 years ago

I added it to the wiki page where we list benchmarks used for the paper.

Neeratyoy commented 2 years ago

To expound on the above: For all tabular Community Benchmarks (primarily the NAS benchmarks) and the new ML Tabular benchmarks, we have the optimum recorded. For reference, you can refer to the wiki page links (comment above) to the experiment repositories (for our paper) where we assumed a certain optimum for the other benchmarks as the best from multiple runs.

tyf0416 commented 2 years ago

Thanks for the responses. Those solved my question.