Incorrect result in computing `MulticlassRecall` macro average when `ignore_index` is specified

JeroenMandersloot commented 8 months ago

🐛 Bug

Specifying the ignore_index argument in MulticlassRecall leads to incorrect results when computing the macro average.

To Reproduce

import torch
from torchmetrics.classification.precision_recall import MulticlassRecall

metric = MulticlassRecall(num_classes=2, ignore_index=0, average="macro")
y_true = torch.tensor([0, 0, 1, 1])
y_pred = torch.tensor([
    [0.9, 0.1],  # Correct
    [0.9, 0.1],  # Correct
    [0.9, 0.1],  # Incorrect
    [0.1, 0.9],  # Correct
])
metric.update(y_pred, y_true)
print(metric.compute())  # Prints 0.25, but I would expect the result to be 0.5

Expected behavior

In the toy example above there are two classes (0 and 1) and two instances of each class. I want to ignore class 0 altogether (hence ignore_index=0), so I would expect the first two predictions to be ignored entirely (since they relate to class 0 instances). Of the remaining two predictions only one is correct, so I would expect the recall score to be 0.5.

However, the actual result is 0.25. It looks as if the recall score for the ignore_index class is simply set to 0 before computing the average, causing the average to be computed as (0 + 0.5) / 2 instead of 0.5 / 1.