search

maxpumperla / hyperas

Keras + Hyperopt: A very simple wrapper for convenient hyperparameter optimization
http://maxpumperla.com/hyperas/
MIT License
2.18k stars 318 forks source link

How can I optimize discrete hyperparameters ? Like batchsize in the range [100,1000]? #214

Closed Jingfei-Liu closed 5 years ago

Jingfei-Liu commented 5 years ago

As the title implies? Would you please help to answer this question?

maxpumperla commented 5 years ago

have a look at randint

Jingfei-Liu commented 5 years ago

I am glad you can reply ! Thank you very much!

hmanz commented 5 years ago

@maxpumperla it appears that randint does not have a lower bound option: https://github.com/hyperopt/hyperopt/wiki/FMin

I suspect we should be using quniform, but that returns floats instead of ints, leading to errors such as this: 

TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64

Any idea what the right approach is? Shouldn't this merge have fixed this issue?

maxpumperla commented 5 years ago

@hmanz just add the lower bound...

hmanz commented 5 years ago

@maxpumperla maybe I'm misunderstanding you, but the documentation for Hyperopt's randint shows that there is no option for a lower bound. If I try to do something like randint(10, 100), I get the error: TypeError: ap_categorical_sampler() got multiple values for argument 'size'

maxpumperla commented 5 years ago

choosing a number between a and b is the same as choosing a number between 0 and b-a and then adding a after choosing:

hp.randint(labels, upper-lower) + lower

i.e. literally adding the lower bound.

hmanz commented 5 years ago

Thanks @maxpumperla, that works! I wasn't sure if you could put anything other than a hyperopt distribution within the double curly braces, but it seems the addition is not problematic.

The only issue with randint is that "there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values." So ideally I would want to use quniform. Any idea why this merge did not solve the float return issue in hyperas? For now what I have done as a workaround is going into the hyperopt pyll_utils.py file and changed scope.float to scope.int in def hp_quniform. This is working so far.