Open phoxis opened 3 years ago
This is a troubling issue, and I don't know how to solve it. The F1 formula is: tp / (tp + 0.5 * (fp+fn)) If tp is 0 the F1 is 0, considering you have an "empty" label, tp is not possible to be different of 0.
What do you suggest, should I use a different F1 formulation for such cases?
An alternative for such cases is compute the measures manually:
> x <- multilabel_evaluate (y, y1, labels = TRUE)
> x$labels
accuracy AUC balacc F1 precision recall TP TN FP FN
y1 0.75 1 0.5 0.8571429 0.75 1 3 0 1 0
y2 1.00 1 1.0 1.0000000 1.00 1 3 1 0 0
y3 1.00 NA NaN 0.0000000 0.00 0 0 4 0 0
y4 1.00 1 1.0 1.0000000 1.00 1 2 2 0 0
y5 1.00 1 1.0 1.0000000 1.00 1 1 3 0 0
I have previously followed as it is implemented in MULAN library. I used the mldr library in R before as well where there was a bug, and I adapted the F-Score from them and used it as follows.
mldr_MacroFMeasure <- function(trueLabels, bipartition) { macroPrecision <- colSums(trueLabels & bipartition) / colSums(bipartition) macroRecall <- colSums(trueLabels & bipartition) / colSums(trueLabels)
macroRecall[is.nan (macroRecall)] <- 1; macroPrecision[is.nan (macroPrecision)] <- 1; temp <- (macroPrecision macroRecall 2 / (macroPrecision + macroRecall)) temp[is.nan(temp)] <- 0; return ( mean (temp)); }
This was a quick fix, but essentially, in case the F-Score is NaN then I consider it 1. It is a quick and dirty trick, but I think when all the targets are zeros and the predictions are all zeros, the F-Score should be 1? A way can be a check if the target is all zeros, then the labels can be flipped (all zeros to ones) to continue with the computation?
Run the following
y
andy
has identical label assignment. But the macro-F1 and other scores are not 1. Probably because the label y3 is all zeros, and therefore it translates to 0. This should be 1 in such a case.