How to use reduction_metric? (Bug?)

[soliverc]

I use metrics from the keras metrics package to track model performance:

import keras_metrics as km

    model.compile(optimizer=params['optimizer'], 
                  loss='binary_crossentropy', 
                  metrics=[km.binary_precision(), km.binary_recall(),km.binary_f1_score()])

When I do this I get the following output:

Epoch 00002: val_f1_score did not improve from 0.56823
Epoch 3/400
14296/14296 [==============================] - 0s 27us/step - loss: 0.6637 - precision: 0.5344 - recall: 0.5724 - f1_score: 0.5527 - val_loss: 0.6678 - val_precision: 0.5309 - val_recall: 0.5832 - val_f1_score: 0.5558

Epoch 00003: val_f1_score did not improve from 0.56823
Epoch 4/400
14296/14296 [==============================] - 0s 26us/step - loss: 0.6574 - precision: 0.5430 - recall: 0.5985 - f1_score: 0.5694 - val_loss: 0.6704 - val_precision: 0.5236 - val_recall: 0.6361 - val_f1_score: 0.5744

etc etc

As you can see there is a val_f1_score in the metrics being displayed. So that's what I've been using for the Scan fuction.

h = Scan(training_features_scaled,
            training_target,
            model=mymodel,
            params=p,
            experiment_name="val_dropout",
            reduction_metric="val_f1_score")

I think I've just discovered that this is not working. I entered akjdfjkahd into reduction_metric and it worked. The model is training now.

So what should I do if I want to use something like val fscore?

This is the only line I can find in the docs:

Reduction Metric: The metric against which optimization is performed (e.g. 'val_acc' or 'fmeasure')

Do I just use val_fmeasure? I can use it, but I can't know if it's working, because this argument accepts any string.

By the way, this library is awesome.

edit: just from reading issues here, I noticed that metrics are available in ta.utils.metrics. So let's say I want to use ta.utils.metrics.matthews.

If I put matthews into reduction_metric, will it work?

Like this:

    model.compile(optimizer=params['optimizer'], 
                  loss='binary_crossentropy', 
                  metrics=[ta.utils.metrics.matthews]) # adding matthews here

h = Scan(training_features_scaled,
            training_target,
            model=mymodel,
            params=p,
            experiment_name="my_experiment",
            reduction_metric="matthews") # add "matthews" here

My model is currently training with reduction_metric="fgsfgsdfgsf" :)

[github-actions[bot]]

Welcome to Talos community! Thanks so much for creating your first issue :)

[mikkokotila]

reduction_metric needs to accept arbitrary strings, because custom metrics might have any imaginable name, so what you are witnessing is expected. But I think there has to be some mechanism to alert when the case is such as here i.e. the input and the available metrics do not correspond.

Meaningful input for reduction_metric is one of the metrics in your experiment, one that you want to use for optimization purpose.

[mikkokotila]

Closing here as resolved. Will create a new ticket for handling the alert.