search

automl / HpBandSter

a distributed Hyperband implementation on Steroids
BSD 3-Clause "New" or "Revised" License
608 stars 109 forks source link

Invalid configuration result #62

Open migmartinho opened 5 years ago

migmartinho commented 5 years ago

Hi, really interesting hyperparameter optimization framework, congrats! I read through the paper and went through the code, but I am still struggling to understand what happens when an invalid configuration is sampled. Is the error from the worker caught and that configuration is assigned infinite loss and the worker advances to a new iteration? If so, that means that for a given number of iterations, it can happen that the initial number of configurations evaluated on a given budget can be less than what was supposed to be, right?

Also, if I set the minimum and maximum budgets to the same value I can switch to normal BO (without Hyperband)?

Thanks!

sfalkner commented 5 years ago

You understood that correctly: If a configuration crashes in the worker, it will count as failed evaluation. The optimizer will go ahead and count this as an evaluation with an infinite loss and sample a new configuration. If many of your configurations fail like that it's hard to predict how many configurations are successfully evaluated. But without knowing any more details, I can't say anything specific.

If you set min_budget equal to max_budget, you do fall back to BO, but full disclaimer: the implementation in hpbandster is not quite as good as the original TPE implementation.