Recall@1 does not return the same results as micro recall (on single-prediction classification)?

[caesar-one]

Hi,

First of all, thank you so much for this useful library!

I have been doing a few very simple tests, to compare the output of this library with the output of scikit-learn's micro averaged metrics (precision and recall). Based on my understanding, with k=1, the results of precision@1 and recall@1 should be the same of micro precision and micro recall, if we have just one label predicted by the model.

This holds for precision@1, but recall@1 returns different results. Here is a minimal example:

import numpy as np
from scipy.sparse import csr_matrix
from xclib.evaluation import xc_metrics
from sklearn.metrics import precision_recall_fscore_support

gold_vecs = np.array([[1, 1, 0, 0], [0, 1, 1, 0], [0, 0, 1, 0]])
pred_vecs = np.array([[0, 0, 1, 0], [0, 0, 1, 0], [0, 0, 1, 0]])

print("sklearn micro:", precision_recall_fscore_support(y_true=gold_vecs, y_pred=pred_vecs, average='micro', zero_division=0))

print("xclib precision@1:", xc_metrics.precision(csr_matrix(pred_vecs), csr_matrix(gold_vecs), k=1)[0])
print("xclib recall@1:", xc_metrics.recall(csr_matrix(pred_vecs), csr_matrix(gold_vecs), k=1)[0])

this outputs:

sklearn micro: (0.6666666666666666, 0.4, 0.5, None)  # the first two values are precision and recall, respectively
xclib precision@1: 0.66666667
xclib recall@1: 0.5

By doing manual verification, I get: 3 fn, 1 fp, 2 tp recall = 2/(2+3) = 0.4

Am I missing something?

[kunaldahiya]

Hi,

Thanks for checking out the library.

In XC setup, we compute the metric for each test point and then take the average - this would correspond to average='samples' in scikit-learn library. The recall@1 for individual samples (in the given example) is 0, 0.5, and 1 respectively and the overall recall@1 is 0.5 as computed by pyxclib.

output as per scikit-learn is: sklearn: (0.6666666666666666, 0.5, 0.5555555555555555, None)