List of metrics documented in Processor but not supported for evaluation. - Githubissues

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

https://farm.deepset.ai

Apache License 2.0

1.74k stars 247 forks source link

List of metrics documented in Processor but not supported for evaluation. #776

Closed johann-petrak closed 3 years ago

johann-petrak commented 3 years ago

The documentation for TextClassificationProcessor states that the parameter metric can take a list of metrics, see https://github.com/deepset-ai/FARM/blob/751b48f0310e11c0e4cbdb95f264e47a587d86e1/farm/data_handler/processor.py#L532

However this is not actually supported at evaluation time in evaluation.metrics.compute_metrics (see https://github.com/deepset-ai/FARM/blob/751b48f0310e11c0e4cbdb95f264e47a587d86e1/farm/evaluation/metrics.py#L74)

Since this is a nice and obvious feature, I think it should be implemented.

Once implemented, it may actually predefined metrics like "acc_f1" superfluous.