for loop to add layers with multiple choices is not working

[virtualdvid]

Hi,

I was looking for examples about how to add multiples layers using a for loop and I got something similar to this code (line 85). Did you notice the for loop is not working how it suppose to be? I mean, I want that depending on the choice in range add some layers with a different choice for each layer but instead, I'm getting yeah multiple layers but all of them with the same choice.

Here an example of the results I got after running minimize:

model = Sequential([
    # first layer out of the loop:
    layers.Conv2D(filters=256, kernel_size=(4, 4), activation='tanh', input_shape=(128, 128, 3)),
    layers.MaxPool2D(pool_size=(2,2)),
    layers.Dropout(0.7139834391535448),

   # second layer in the loop after for _ in range(2):
    layers.Conv2D(filters=128, kernel_size=(3, 3), activation='relu'), # << same filter and kernel
    layers.MaxPool2D(pool_size=(2,2)),
    layers.Dropout(0.48363463144260765), # << same dropout

    # third layer in the loop
    layers.Conv2D(filters=128, kernel_size=(3, 3), activation='relu'), # << same filter and kernel
    layers.MaxPool2D(pool_size=(2,2)),
    layers.Dropout(0.48363463144260765), # << same dropout

    layers.Flatten(),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.7692559015213345),
    layers.Dense(5, activation='sigmoid', name='predict')

What am I doing wrong? or shouldn't I use a loop at all? Thanks!

Regards, David Molina

[maxpumperla]

@virtualdvid unfortunately, it doesn't work as you want. hyperas gets translated to hyperopt. hyperopt first samples from the search space and then plugs it into the resp. places. If you wrap that in a loop, it will still only sample once and plug the same value in n times. so it's intended behaviour, not a bug.

If you want this built-in I need to think about it a bit. You can do it with plain hyperopt by parameterizing the search space, but it's not necessarily fun either. The quick win is to unpack the loop :/

[virtualdvid]

@maxpumperla Thanks for your answer! yeah that's what I thought. After playing daily for a month with hyperas there are several things I would like to have lol. Anyway it's fine, I did the unpack and it's working good. I was just confused with this code. I was wondering if that worked for @deKeijzer what am I doing wrong?. Good to know it is an expected behavior and not a bug. Thanks again!

Regards, David Molina

[deKeijzer]

@virtualdvid Could you post your exact code? The one in your OP is a bit confusing, since it's not correct. Although i do understand what you're trying to do.

Basically what i did in my code is initialize a sequential model:

model = Sequential()

And add layers like you normally would.

model.add(Conv2D(...)) ...

Say you'd want to add 5 times the exact same layer configuration (without hyperopt), you could write a loop like this:

model = Sequential()

for _ in range( 5 ):
     model.add(Conv2D(filters=(4,4), 
                 kernel_size= (4, 4), 
                 padding='same',
                kernel_initializer='TruncatedNormal'))
      model.add(BatchNormalization())
      model.add(LeakyReLU())
      model.add(Dropout(0.5)) 

Now to apply hyperopt to this, just add the {{...}} where you would like. For example:

     model = Sequential()

    ks1_first = {{choice([2, 3, 4, 5, 8, 10])}}
    ks1_second = {{choice([2, 3, 4, 5, 8, 10])}}

    ks2_first = {{choice([2, 3, 4, 5, 8, 10])}}
    ks2_second = {{choice([2, 3, 4, 5, 8, 10])}}

    model.add(Conv2D(filters=( {{choice([1, 2, 3, 4, 5, 8, 12])}} ), 
                     kernel_size=(ks1_first, ks1_second),
                     input_shape=input_shape, 
                     padding='same',
                     kernel_initializer='TruncatedNormal'))
    model.add(BatchNormalization())
    model.add(LeakyReLU())
    model.add(Dropout( {{uniform(0, 1)}} ))

Without seeing your original code, i would suggest trying to get it to work properly without hyperopt first. Once the loops for the layer configurations are working, apply hyperopt.

[virtualdvid]

Hi @deKeijzer, I'm sorry for my code. That's the one I use to replace the final result I obtained using hyperas to train the model again outside hyperas. Here the loop I was trying to approach:

Did two test one with the {{ .. }} variables inside the loop:

model = Sequential()

... initial layers

range_0 = {{choice([2, 3])}}
for _ in range(range_0):
    filters = {{choice([32, 64, 128, 256, 512])}}
    kernel_sizes = {{choice([3, 4])}}
    activations = {{choice(['relu', 'tanh'])}}
    model.add(layers.Conv2D(
            filters=filters,
            kernel_size=kernel_sizes,
            activation=activations
        )
    )

The other one outside the loop:

model = Sequential()

... initial layers

filters = {{choice([32, 64, 128, 256, 512])}}
kernel_sizes = {{choice([3, 4])}}
activations = {{choice(['relu', 'tanh'])}}
range_0 = {{choice([2, 3])}}
for _ in range(range_0):
    model.add(layers.Conv2D(
            filters=filters,
            kernel_size=kernel_sizes,
            activation=activations
        )
    ) 

Same result in both. Sadly as @maxpumperla answered once the space is generated it doesn't change the choices in the loop. At the end, I got three or two layers exactly the same which is annoying lol.

[deKeijzer]

@virtualdvid Is the behavior the same if you put the {{...}} inside of range({{...}}) instead of range(variable)? If that does not change anything, it might as well be that i never noticed this behavior.

[virtualdvid]

@deKeijzer it works exactly the same, I just put it into a variable to get a "custom" name, you know hyperas named things in a confusing way.

[deKeijzer]

@virtualdvid The last thing you could try is running my code and try to figure out if that works. A simple git clone should do, as everything (including the data) is in the repository. Unfortunately i'm unable to test the code myself due to some trouble with my current GPU setup. I thought the loops were working, but i might as well have not noticed it to be faulty. The code had been written in quite the hurry... Sorry to not be able to help you any further.

Good luck!