search

HIPS / Spearmint

Spearmint Bayesian optimization codebase
Other
1.55k stars 329 forks source link

int_to_unit and float_to_unit break if max == min #17

Open JesseLivezey opened 9 years ago

JesseLivezey commented 9 years ago

In spearmint/tasks/base_task.py, the functions int_to_unit and float_to_unit assume that vmax > vmin. When running experiments with trivial parameter ranges (vmax == vmin), this breaks when I think the reasonable behavior is to just always choose the one value.

https://github.com/HIPS/Spearmint/blob/92d6b5d6e7bc008c5a68df52c2d27611cce69ba4/spearmint/tasks/base_task.py#L372

JasperSnoek commented 9 years ago

I think it is silly to have the code break when it would be much better to return an error message. However, I can't think of a good reason why anyone would set vmax == vmin besides by mistake. Perhaps we should just add a helpful error message here?

JesseLivezey commented 9 years ago

I don't have a good reason, but I do have a bad reason! (laziness)

One example would be that I'm trying to optimize multiple hyperparameters at once. Later, I want to run the same optimization, but fix one of the parameters rather than optimizing over it. If I can set vmin == vmax and just get the trivial result for that parameter, then the rest of the code can stay the same. If I have to remove it from the list of hyperparameters, I then have to insert the parameter somewhere else in the code which is just more work for me.

I tried hacking it in by having a check here: https://github.com/HIPS/Spearmint/blob/3b0f241799d23344956c26ba1d05cfe213aef0d8/spearmint/tasks/base_task.py#L375 and then setting unit to zero if vmin == vmax. This runs, but I don't understand the code well enough to know if this will break any of the GP stuff down the line. Do you think something like this would work?

I can make a PR with at least a better warning.

JasperSnoek commented 9 years ago

Hi Jesse, yeah I understand that. It would require building in some logic to have those variables bypass the GP, since otherwise there would be unnecessary computation happening (in computing covariances between these zeros). I think your trick would work though, so you can do it in your own branch if you like.

I'd rather leave that kind of bookkeeping up to the user than try to build it in to Spearmint (or perhaps a user made wrapper around Spearmint to do neat user-friendly bookkeeping, etc.) @mgelbart, what do you think?

Definitely a better warning would be great :-) I added a more useful assert. Thanks!

Jasper

On Mon, May 11, 2015 at 7:24 PM, Jesse Livezey notifications@github.com wrote:

I don't have a good reason, but I do have a bad reason! (laziness)

One example would be that I'm trying to optimize multiple hyperparameters at once. Later, I want to run the same optimization, but fix one of the parameters rather than optimizing over it. If I can set vmin == vmax and just get the trivial result for that parameter, then the rest of the code can stay the same. If I have to remove it from the list of hyperparameters, I then have to insert the parameter somewhere else in the code which is just more work for me.

I tried hacking it in by having a check here:

https://github.com/HIPS/Spearmint/blob/3b0f241799d23344956c26ba1d05cfe213aef0d8/spearmint/tasks/base_task.py#L375 and then setting unit to zero if vmin == vmax. This runs, but I don't understand the code well enough to know if this will break any of the GP stuff down the line. Do you think something like this would work?

I can make a PR with at least a better warning.

— Reply to this email directly or view it on GitHub https://github.com/HIPS/Spearmint/issues/17#issuecomment-101073230.

mgelbart commented 9 years ago

I think if you remove the parameter from the config entirely then everything should be ok, but as you said your code would need to know that this parameter is not coming in. I agree with Jasper's assessment.

Mike

JesseLivezey commented 9 years ago

Thanks for the feedback!

I'm probably also going to write something that just filters out parameters with trivial ranges and adds them back in at the end. Once I get it working, I'll post it here in case it's useful for others.

JasperSnoek commented 9 years ago

That sounds great!

Cheers,

Jasper

On Wed, Jun 3, 2015 at 8:23 PM, Jesse Livezey notifications@github.com wrote:

Thanks for the feedback!

I'm probably also going to write something that just filters out parameters with trivial ranges and adds them back in at the end. Once I get it working, I'll post it here in case it's useful for others.

— Reply to this email directly or view it on GitHub https://github.com/HIPS/Spearmint/issues/17#issuecomment-108656140.