search

automl / HPOBench

Collection of hyperparameter optimization benchmark problems
Apache License 2.0
136 stars 35 forks source link

Some confusions about nasbenchmark_201. #178

Open fze0012 opened 1 year ago

fze0012 commented 1 year ago

For different task numbers, how can I know the best results for canculating the simple or inference regret?

Neeratyoy commented 1 year ago

The best score is known only for the tabular benchmarks. For the nn benchmarks the following should work:

from hpobench.benchmarks.ml import TabularBenchmark

b = TabularBenchmark(model="nn", task_id=31)

b.global_minimums
fze0012 commented 1 year ago

https://github.com/automl/HPOBench/blob/47bf141f79e6bdfb26d1f1218b5d5aac09d7d2ce/hpobench/benchmarks/nas/nasbench_201.py#L108-L124

In this file, what is the mean of the prefix ori e.g. ori-test?

Neeratyoy commented 1 year ago

Hi,

This docstring is borrowed from the NASBench-201 paper release and thus the actual details can be found here. This is likely to indicate the numbers on the original test set.

I shall close this issue for now as there is nothing about HPOBench here. Please feel free to reopen or ask any further queries.

fze0012 commented 1 year ago

https://github.com/automl/HPOBench/blob/47bf141f79e6bdfb26d1f1218b5d5aac09d7d2ce/hpobench/benchmarks/nas/nasbench_201.py#L262-L264 For test_accuracies and test_losses, why the valid_key is used rather than the test_key?

Neeratyoy commented 1 year ago

Thanks for raising this. Would you like to do a PR with this fix?

You can refer to this and this and use the local version for testing. Once merged, we can upload a new container with this fix.