Open blademoon opened 9 months ago
My current hypothesis is that the problem is in the data type returned by the metrics. Metrics with no problem combining returns the dictionary:
m_accuracy = evaluate.load("accuracy")
m_recall = evaluate.load("recall")
m_precision = evaluate.load("precision")
m_f1 = evaluate.load("f1")
pred = [1,0,1]
ref = [1,1,1]
result_accuracy = m_accuracy.compute(predictions=pred, references=ref)
result_recall = m_recall.compute(predictions=pred, references=ref)
result_precision = m_precision.compute(predictions=pred, references=ref)
result_f1 = m_f1.compute(predictions=pred, references=ref)
print("Accuracy:", type(result_accuracy), result_accuracy)
print("Recall:", type(result_recall), result_recall)
print("precision:", type(result_precision), result_precision)
print("F1:", type(result_f1), result_f1)
Output:
Accuracy: <class 'dict'> {'accuracy': 0.6666666666666666}
Recall: <class 'dict'> {'recall': 0.6666666666666666}
precision: <class 'dict'> {'precision': 1.0}
F1: <class 'dict'> {'f1': 0.8}
Metrics that have problems (such as WER, CER) return a value when trying to combine:
m_wer = evaluate.load("wer")
m_cer = evaluate.load("cer")
pred = ["test", "Test"]
ref = ["TEST", "Test"]
result_wer = m_wer.compute(predictions=pred, references=ref)
result_cer = m_cer.compute(predictions=pred, references=ref)
print("WER:", type(result_wer), result_wer)
print("CER:", type(result_cer), result_cer)
Output:
WER: <class 'float'> 0.5
CER: <class 'float'> 0.5
Naturally float
has no keys
attribute...
It is possible to make changes to the code of these files:
https://github.com/huggingface/evaluate/blob/main/metrics/wer/wer.py#98 https://github.com/huggingface/evaluate/blob/main/metrics/wer/wer.py#106
https://github.com/blademoon/evaluate/blob/main/metrics/cer/cer.py#140 https://github.com/blademoon/evaluate/blob/main/metrics/cer/cer.py#159
may solve the problem. Hypothetical, you just need to "wrap" the return value in a dictionary with the appropriate key.
Hello. Reproduction code:
Code output:
Python verion:
Library versions: