Closed Neeratyoy closed 5 years ago
thanks for the PR. Could you add a catch inside NASCifar10 such that if you set the budget to something smaller than 108 in the objective function it throws an error when you only loaded the full budget dataset?
Hope an assert is adequate.
These changes do not alter the current behaviour of the classes.
When scheduling multiple runs, the benchmark needs to be initialized every time. Especially for the full CIFAR benchmark, it eats up a fair bit of time which multiplicatively increases with the number of runs.
Adding a
reset_tracker
function to the class which does the job of__init__
without the data loading.Also for the CIFAR, adding an option to specify if it is not a multi-fidelity optimization task and therefore load the lighter dataset (_nasbenchonly108.tfrecord) with the fixed epoch budget evaluations.