In section 6 of "Panoptic Quality" paper, under "Human annotations", it is mentioned that PQ is symmetric, i.e. the order of ground truth and predictions is unimportant. In practice this does not seem to be the case, for example:
One of the reasons seems to be the special treatment of VOID class, for example predictions are not counted as false positives when their overlap with ground truth VOID is bigger than 0.5 (section 4.2 in the paper, under "Void labels"). I think it enforces one interpretation of VOID, meaning "not labeled", when ignoring such areas would make sense. Alternative would be "not known class", in which case marking it with known class would count as false positive. The second interpretation is more common IMHO and would retain the symmetric property of the metric.
I suggest to change the interpretation of VOID to make the metric symmetric, or add it as a command-line option.
In section 6 of "Panoptic Quality" paper, under "Human annotations", it is mentioned that PQ is symmetric, i.e. the order of ground truth and predictions is unimportant. In practice this does not seem to be the case, for example:
One of the reasons seems to be the special treatment of VOID class, for example predictions are not counted as false positives when their overlap with ground truth VOID is bigger than 0.5 (section 4.2 in the paper, under "Void labels"). I think it enforces one interpretation of VOID, meaning "not labeled", when ignoring such areas would make sense. Alternative would be "not known class", in which case marking it with known class would count as false positive. The second interpretation is more common IMHO and would retain the symmetric property of the metric.
I suggest to change the interpretation of VOID to make the metric symmetric, or add it as a command-line option.