automl / nas_benchmarks

BSD 3-Clause "New" or "Revised" License
91 stars 25 forks source link

Faster benchmark initialization for multiple runs #18

Closed Neeratyoy closed 5 years ago

Neeratyoy commented 5 years ago

These changes do not alter the current behaviour of the classes.

When scheduling multiple runs, the benchmark needs to be initialized every time. Especially for the full CIFAR benchmark, it eats up a fair bit of time which multiplicatively increases with the number of runs.

Adding a reset_tracker function to the class which does the job of __init__ without the data loading.

Also for the CIFAR, adding an option to specify if it is not a multi-fidelity optimization task and therefore load the lighter dataset (_nasbenchonly108.tfrecord) with the fixed epoch budget evaluations.

aaronkl commented 5 years ago

thanks for the PR. Could you add a catch inside NASCifar10 such that if you set the budget to something smaller than 108 in the objective function it throws an error when you only loaded the full budget dataset?

Neeratyoy commented 5 years ago

Hope an assert is adequate.