Is the calculation of mean IoU reasonable?

[MichaleWong]

❓ Questions & Help

in the function of mean_iou, I mean the code "iou[torch.isnan(iou)] = 1", is it reasonable,? when I test my model, some type has a bad segmentation, like board on s3dis dataset, it should get a very low score, but get a high score(0.95), in IOU, there are many ‘nan’ value，the code change them to 1, so it's very high score. so is it reasonable to change 'nan' to 1?

def mean_iou(pred, target, num_classes, batch=None): r"""Computes the mean intersection over union score of predictions. Args: pred (LongTensor): The predictions. target (LongTensor): The targets. num_classes (int): The number of classes. batch (LongTensor): The assignment vector which maps each pred-target pair to an example. :rtype: :class:Tensor """ i, u = intersection_and_union(pred, target, num_classes, batch) iou = i.to(torch.float) / u.to(torch.float) iou[torch.isnan(iou)] = 1 //in multiple class segmentation task, some types are not include in the test data iou = iou.mean(dim=-1) return iou

[rusty1s]

I believe there should be only NaN values in case the ground-truth label does not appear in your example (in case union = 0). Therefore, I feel that it is reasonable to set it to 1.

[GericoVi]

I am also curious about this implementation. Wouldn't giving label types which don't appear (i.e. union=0) an iou value of 1, artificially inflate the metric? Essentially giving samples with less semantic types a bias, since in this case if a ground truth label is not present it gets the maximum value (1)?

Wouldn't it be better to discount the nans instead? With numpy.nanmean for example (or torch.nanmean in pytorch 1.10), or iou[~iou.isnan()].mean()?

What are your thoughts on this? Is there mathematical reasoning that makes the 1 value acceptable in this case that I've missed?

Ps. Excellent library, I appreciate your work on this.

[rusty1s]

How about we add a flag to let the user decide on how to incorporate NaN values? How are other libraries handling thise use-case? Adding support for torch.nanmean() sounds great IMO.

[GericoVi]

That sounds like a good idea. The only comparable library that I know which also handles this use-case is torch-points3d. They only take into account the non NaN values when doing the mean - using a mask index. (See get_average_intersection_union() and `get_intersection_union_per_class() in https://github.com/nicolas-chaulet/torch-points3d/blob/master/torch_points3d/metrics/confusion_matrix.py).

While torch.nanmean() would probably be the ideal solution, this was only added in pytorch 1.10. Maybe the mask option would be better for now for backwards compatibility? (Torch docs states that "torch.nanmean(a) is equivalent to torch.mean(a[~a.isnan()])" - https://pytorch.org/docs/stable/generated/torch.nanmean.html)

[rusty1s]

Yes, we can use torch.mean(a[~a.isnan()]) for this as well. Do you want to integrate it?

[GericoVi]

Yes I could do that. If the below is acceptable, I will do a pull request.

def mean_iou(pred, target, num_classes, batch=None, omitnans=False):
    r"""Computes the mean intersection over union score of predictions.

    Args:
        pred (LongTensor): The predictions.
        target (LongTensor): The targets.
        num_classes (int): The number of classes.
        batch (LongTensor): The assignment vector which maps each pred-target
            pair to an example.

    :rtype: :class:`Tensor`
    """
    i, u = intersection_and_union(pred, target, num_classes, batch)
    iou = i.to(torch.float) / u.to(torch.float)

    if omitnans:
        iou = iou[~iou.isnan()].mean()
    else:
        iou[torch.isnan(iou)] = 1
        iou = iou.mean(dim=-1)

    return iou

For reference, on a trained PointNet++ segmentation model, an example from the airplane class of ShapeNet gives an mIoU of 0.85191 with omitnans = True and 0.98815 with omitnans = False. The former agrees with published results better.