0 valid mIOU for unknown class

[anirudh-chakravarthy]

Hi,

While I'm trying to use this repo to train a network, I observe the following validation performance:

car : 93.16%
bicycle : 15.65%
motorcycle : 27.96%
truck : 42.25%
bus : 0.00%
person : 61.78%
bicyclist : 44.78%
motorcyclist : 0.00%
road : 91.67%
parking : 32.82%
sidewalk : 76.48%
other-ground : 0.01%
building : 88.26%
fence : 48.89%
vegetation : 82.49%
trunk : 61.60%
terrain : 58.30%
pole : 62.96%
traffic-sign : 44.74%

As per my understanding, bus is one of the held-out classes, so I'm a bit concerned to see 0% at the end of training. While I'm not 100% certain, could it have something to do with this line?

[Jun-CEN]

Hi,

train_cylinder_asym_ood_final.py is for the first stage of my method, which aims to detect unknown classes. In this case, we will not calculate the IoU of this class, since the evaluation of unknown class detection is a binary classification (known/unknown) and threshold-independent problem. We use AUROC to evaluate the unknown class detection problem. I suggest you to see the definition of AUROC and you will understand.

[anirudh-chakravarthy]

Thanks for your response. I have 3 follow-up questions:

1. I am wondering whether it's possible to compute IoU over the new class as well. Given Equation 2 from the paper, this seems like a natural extension to perform evaluation?
2. In the OSeg stage, it seems like the network is not supervised on "other-vehicle" (based on this line). Is this understanding correct? And if so, does this imply that OSeg training only uses the synthesized objects as supervision (and what is the class index for these objects since 0 is ignored during loss computation / a code pointer where this happens would be great)?
3. What is the value of $\lambda_{thresh}$ used in your implementation (as stated in Eqn 10) for Semantic-KITTI?