Key error: 'val_fmeasure'

[ghost]

Hi! I am trying to output the best model of talos with the highest validation f-measure. I get a Key error when I do: print("Highest val_fmeasure: {}".format(report.high(metric="val_fmeasure"))); I am following the example in the documentation.

I have just downloaded the last version of talos, so I am up to date. The (toy) hyperparameter optimization runs successfully to 100%, with the following results and then I get the key error with the following trace. The scan_object does not contain val_fmeasure. Do I have to use probabilistic reduction with fmeasure as reduction metric in order to be able to print the val_fmeasure?

Hyperparameter results : round_epochs val_loss val_acc loss acc batch_size epochs 0 16 0.633982 0.68595 0.030584 0.99446 16 10 Best parameters for 'val_acc': [[16 10 0]]

Traceback: File "/home/konstantina/projects/myproject/semeval/d_classification/funcs.py", line 379, in run_model print("Highest val_fmeasure: {}".format(report.high(metric="val_fmeasure"))); File "/home/konstantina/anaconda3/envs/my_env/lib/python3.6/site-packages/talos/commands/reporting.py", line 31, in high return max(self.data[metric]) File "/home/konstantina/anaconda3/envs/my_env/lib/python3.6/site-packages/pandas/core/frame.py", line 2688, in getitem return self._getitem_column(key) File "/home/konstantina/anaconda3/envs/my_env/lib/python3.6/site-packages/pandas/core/frame.py", line 2695, in _getitem_column return self._get_item_cache(key) File "/home/konstantina/anaconda3/envs/my_env/lib/python3.6/site-packages/pandas/core/generic.py", line 2489, in _get_item_cache values = self._data.get(item) File "/home/konstantina/anaconda3/envs/my_env/lib/python3.6/site-packages/pandas/core/internals.py", line 4115, in get loc = self.items.get_loc(item) File "/home/konstantina/anaconda3/envs/my_env/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'val_fmeasure'

[mikkokotila]

Your example does not seem to have fmeasure as one of the metrics. Add that your metrics in the Keras model, and then you will have it. In your case the code you are trying to execute would work if you replace val_fmeasure with val_acc.

First:

from talos.metrics.keras_metrics import f1score

...and then in your Keras input model:

model.compile(... metrics=['acc', f1score] ...)

Closing here, feel free to open new issue if anything.

[ghost]

Thanks for the reply.

from talos.metrics.keras_metrics import f1score does not work out. There is only fmeasure_acc and fbeta_score_acc that I can import. Which one should I choose?

Regarding the Keras model, I suppose you mean I should you the string f1score and not the actual function, right? So, like this: current_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc', 'f1score']); I just tried it and I get : ValueError: Unknown metric function:f1score

[ghost]

Ok, I think I figured it out.

from talos.metrics.keras_metrics import fmeasure_acc, fbeta_score_acc

def f1_score(y_true, y_pred): .... return f1score

if using own f-measure function: f1_score

`current_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc', f1_score]);

print("Highest f1_score: {}".format(report.high(metric="f1_score")));`

if using talos f-measure function: fmeasure_acc

`#current_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc', fmeasure_acc]);

print("Highest val_fmeasure_acc: {}".format(report.high(metric="val_fmeasure_acc")));`

if using talos f-beta function: fbeta_score_acc

current_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc', fbeta_score_acc]); print("Highest val_fbeta_score_acc: {}".format(report.high(metric="val_fbeta_score_acc")));

Could you please update the documentation https://autonomio.github.io/docs_talos/#reporting ? It is confusing, because the _acc prefixes are not mentioned. Thanks!

[mikkokotila]

I have just downloaded the last version of talos, so I am up to date

This fooled me, as I understood you were on V.0.6 or higher, but you are on v.0.5. The names changed as a part of major cleanup in that change.