Error when tuning number of hidden layers

[kateruksha]

Hi!

I faced an error trying when adding hidden_layers to a list of parameters. To replicate it, I followed Breast Cancer study, just had to change "shape" to "shapes" in the parameter list.

def breast_cancer_model(x_train, y_train, x_val, y_val, params):
    model = Sequential()
    model.add(Dense(10, input_dim=x_train.shape[1],
                    activation=params['activation'],
                    kernel_initializer='normal'))

    model.add(Dropout(params['dropout']))

    hidden_layers(model, params, 1)

    model.add(Dense(1, activation=params['last_activation'],
                    kernel_initializer='normal'))

    model.compile(loss=params['losses'],
                  optimizer=params['optimizer'](lr=lr_normalizer(params['lr'],params['optimizer'])),
                  metrics=['acc', fmeasure])

    history = model.fit(x_train, y_train, 
                        validation_data=[x_val, y_val],
                        batch_size=params['batch_size'],
                        epochs=params['epochs'],
                        verbose=0)

    return history, model

p = {'lr': (0.5, 5, 10),
     'first_neuron':[4, 8, 16, 32, 64],
     'hidden_layers':[0, 1, 2],
     'batch_size': (2, 30, 10),
     'epochs': [150],
     'dropout': (0, 0.5, 5),
     'weight_regulizer':[None],
     'emb_output_dims': [None],
     'shapes':['brick','long_funnel'],
     'optimizer': [Adam, Nadam],
     'losses': [ binary_crossentropy],
     'activation':[relu, elu],
     'last_activation': [None]}

t = ta.Scan(x=x,
            y=y,
            model=breast_cancer_model,
            params=p,
            dataset_name='breast_cancer',
            experiment_no='1')

This returns an error: IndexError: list index out of range

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-16-109ab999a0ed> in <module>
      4             params=p,
      5             dataset_name='breast_cancer',
----> 6             experiment_no='1')

~/.local/lib/python3.5/site-packages/talos/scan/Scan.py in __init__(self, x, y, params, model, dataset_name, experiment_no, x_val, y_val, val_split, shuffle, round_limit, grid_downsample, random_method, seed, search_method, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, reduce_loss, last_epoch_value, clear_tf_session, disable_progress_bar, print_params, debug)
    168         # input parameters section ends
    169 
--> 170         self._null = self.runtime()
    171 
    172     def runtime(self):

~/.local/lib/python3.5/site-packages/talos/scan/Scan.py in runtime(self)
    173 
    174         self = scan_prepare(self)
--> 175         self = scan_run(self)

~/.local/lib/python3.5/site-packages/talos/scan/scan_run.py in scan_run(self)
     16                      disable=self.disable_progress_bar)
     17     while len(self.param_log) != 0:
---> 18         self = scan_round(self)
     19         self.pbar.update(1)
     20     self.pbar.close()

~/.local/lib/python3.5/site-packages/talos/scan/scan_round.py in scan_round(self)
     30     # fit the model
     31     try:
---> 32         _hr_out, self.keras_model = ingest_model(self)
     33     except TypeError as err:
     34         if err.args[0] == "unsupported operand type(s) for +: 'int' and 'numpy.str_'":

~/.local/lib/python3.5/site-packages/talos/model/ingest_model.py in ingest_model(self)
      8                       self.x_val,
      9                       self.y_val,
---> 10                       self.round_params)

<ipython-input-7-7bad7210c65c> in breast_cancer_model(x_train, y_train, x_val, y_val, params)
     10 
     11     # if we want to also test for number of layers and shapes, that's possible
---> 12     hidden_layers(model, params, 1)
     13 
     14     # then we finish again with completely standard Keras way

~/.local/lib/python3.5/site-packages/talos/model/layers.py in hidden_layers(model, params, last_neuron)
     68     for i in range(params['hidden_layers']):
     69 
---> 70         model.add(Dense(layer_neurons[i],
     71                         activation=params['activation'],
     72                         use_bias=use_bias,
IndexError: list index out of range

Thank you!

[Peon26]

Hi, I have also noticed the same inconsistency, but looking into the network_shape.py clarified that there are only 4 options: 'brick', 'funnel', 'triangle' and a float number (smaller than 1). Passing a string or value that is not defined in that list, will return an empty list, and in layers.py, then layer_neurons = [] which as you can see causes problem:

[...]
    layer_neurons = network_shape(params, last_neuron)

    for i in range(params['hidden_layers']):

        model.add(Dense(layer_neurons[i], 
[...]

So if you remove 'long_funnel' and add 'funnel' your code should work. But I still don't know what does shape argument in the dictionary represent (shapes is a representation which would be used to calculate the number of neurons in each hidden layer accordingly, but shape hasn't been used anywhere), I am using talos.__version__ = '0.4.9'. However, in 0.6.0 there are changes in the arguments' names which would probably cause a lot of problems when one wants to use the newer version.

[mikkokotila]

Thanks @Peon26, that's right. The documentation is not up-to-date, and will be updated to match v.0.6.0 once that makes it's way to pypi.

However, in 0.6.0 there are changes in the arguments' names which would probably cause a lot of problems when one wants to use the newer version.

While there are notable changes and improvements, the Scan argument changes are going to be very straightforward.

Closing here.