search

automl / nas_benchmarks

BSD 3-Clause "New" or "Revised" License
91 stars 25 forks source link

Handling of objective function evaluation in Regularized Evolution script #16

Closed Neeratyoy closed 5 years ago

Neeratyoy commented 5 years ago

Hi,

This is wrt fcnet_tabular_benchmarks.

The Regularized Evolution script evaluates accuracy by subtracting the objective_function() evaluation from 1.

However, for certain benchmarks the MSE returned is greater than 1, which can make the accuracy negative. Is this evaluation correct? Accuracies are computed here and here.

One way to verify this may be so:

b = FCNetSliceLocalizationBenchmark(data_dir) cs = b.get_configuration_space()

for i in range(1000):      config = cs.sampleconfiguration()      value, = b.objectivefunction(config)      if value > 1:          print(config)          break for i in range(6):     value, = b.objective_function(config)     print(1 - value)

aaronkl commented 5 years ago

Good catch. It should be probably just -y instead of 1-y since RE is maximizing, even better would be to change it completely such that it minimizes the objective function. However, I don't think that changes anything on the results. Could you send a PR to fix that?