LSTM optimization

[reinert]

I'm trying to optimize a simple LSTM network but I'm not quite sure if I'm doing it right.

p = {'first_neuron': [64, 128, 256, 320, 480],
     'batch_size': [12, 18, 24, 40, 60],
     'epochs': [100],
     'dropout': (0, 0.1, 10),
     'weight_regulizer': [None],
     'emb_output_dims': [None],
     'optimizer': ['nadam'],
     'losses': ['mse']}

def cp_model(x_train, y_train, x_val, y_val, params):

    model = Sequential()
    model.add(CuDNNLSTM(params['first_neuron'], input_shape=(x_train.shape[1], x_train.shape[2])))
    model.add(Dropout(params['dropout']))
    model.add(Dense(1))

    model.compile(loss=params['losses'], optimizer=params['optimizer'])    

    out = model.fit(x_train, y_train,
                    batch_size=params['batch_size'],
                    epochs=params['epochs'],
                    verbose=0,
                    callbacks=[live()],
                    validation_data=[x_val, y_val])

    return out, model

h = ta.Scan(x, y,
            params=p,
            model=cp_model,
            dataset_name='cp',
            experiment_no='1',
            grid_downsample=.1,
            val_split=.2,
            reduction_metric='val_loss',
            reduce_loss=True,
            shuffle=False)

This code works fine. But I want to retrive the best 3 models (according to the val_loss) and see their hyperparameters. How can I do this?

I did the following:

e = ta.Evaluate(h)
e.evaluate(x, y, metric='val_loss', asc=True, mode='regression')

but it returns me an array with 5 values that I don't know what they are.

[reinert]

I'm exploring the Reporting function, and seeing the results. Altough Reporting seems to be coded more for Classification problems, instead of Regression where we want to minimize loss instead of maximizing accuracy.

The values presented by the Report are the values of last iteration in each experiment? I would like to get the best value of each experiment, how can I achieve it?

[reinert]

Ok, I found that getting the best epoch is the default behavior. If I want to get the last epoch then I can just set last_epoch_value=True.

Now I want to know if I can customize the score of each epoch. Instead of getting the best val_loss, I want to get the best sum of val_loss and loss. Is it possible?

[reinert]

We could just add a function parameter to Scan so one can define what value to report here in results.py#L52.

The function receives the history and returns the value to report.

I'll get into it ASAP.

[toddpi314]

@reinert You can start by adding a custom metric that defines the minimization logic.. something like this:

    def linear_regression_equality(y_true, y_pred):
        ...
        return -1/1

    model.compile(
        ...
        metrics=[
            linear_regression_equality
        ]
    )

Then, when talos reports back, just do a look-up on the response dataframe:

        report = ta.Reporting(scan)

        typed_report_data = report.data.convert_objects(convert_numeric=True)
        typed_report_data = typed_report_data.loc[
            typed_report_data['val_linear_regression_equality']
            <= typed_report_data['val_linear_regression_equality'].min()]
        typed_report_data = typed_report_data.loc[
            typed_report_data['linear_regression_equality']
            <= typed_report_data['linear_regression_equality'].min()]
        typed_report_data = typed_report_data.loc[
            typed_report_data['loss']
            <= typed_report_data['loss'].min()]
        typed_report_data = typed_report_data.loc[
            typed_report_data['val_loss']
            <= typed_report_data['val_loss'].min()]

        best_model_id = typed_report_data.iloc[0].name - 1
        best_model = ta.utils.best_model.activate_model(scan, best_model_id)

Hope that helps...

[mikkokotila]

Now I want to know if I can customize the score of each epoch. Instead of getting the best val_loss, I want to get the best sum of val_loss and loss. Is it possible?

It's possible, the history object for each permutation is stored in the scan object. Check it out and then go from there.

We could just add a function parameter to Scan so one can define what value to report here in results.py#L52.

Sounds like an interesting idea :)

[reinert]

@toddpi314 you really helped me! thank you!!

@mikkokotila I'll notify you whenever I get into it.

[reinert]

what is the latest branch so I can implement over it? I noticed 'params-api-test' doesn't have the results.py file anymore.

[mikkokotila]

'params-api-test' is the branch. There is now a sub-module (folder) logging and you will find anything related with logging/results there.

[reinert]

It's possible, the history object for each permutation is stored in the scan object.

@mikkokotila, where exactly?

[mikkokotila]

It's self.round_history

[reinert]

It's self.round_history

Ok, it's present in params-api_test branch, but not in current pip version.

I'm working with this branch now, but the Scan object does not accept the grid_downsample anymore . How can I inform it?

[mikkokotila]

Sorry for the delay in getting back to this. It's fraction_limit now. Everything that creates a limit of some sort, ends with _limit

For reference:

fraction_limit
round_limit
time_limit
boolean_limit

[mikkokotila]

closing here. Feel free to open new issue/s as needed.