Question about visualization and energy threshold

[misraya]

Hi,

Great work on unknown detection!

I am trying to use your pretrained model (on BDD) to evaluate and visualize on multiple datasets, including COCO and nuImages. From this line, I understand that you suggest 8.868 to use as the energy threshold. However, when I use it, all objects are labeled as unknown, which shouldn't be the case in my opinion. I looked it up by using the cutoff value to index the thresholds array (with no other changes to the code), it turns out to be 3.5726, but when I use it no object is labeled as unknown. Considering both cases, I feel like it should be somewhere in between, but I'm not sure what other way I can calculate it.

Could you please tell me how do you calculate the threshold value, and what value you used for the visualizations?

Thanks a lot.

[d12306]

hi, @misraya , you can calculate the score threshold when the true positive rate of ID samples is at 95%. In the implementation level, the score threshold can be obtained by "thres = thresholds[cutoff]" below this line.

https://github.com/deeplearning-wisc/stud/blob/c356bbec749ef74c2d338df742d2d88f9dc776d3/tools/metric_utils.py#L84

[misraya]

that's exactly what I did using your pre-trained model and it returns 3.5726, which when used for visualization, doesn't output any unknown detections as in the ones shown in the paper. I'm pasting some examples from coco and nuImages for reference to compare to Figures 7 and 8.

I'm wondering if the value 3.5726 is wrong or is there something else I'm missing? what threshold did you use? thanks in advance!

[d12306]

hi, @misraya , sorry for the late response!

Can you uncomment these two lines ? https://github.com/deeplearning-wisc/stud/blob/c356bbec749ef74c2d338df742d2d88f9dc776d3/src/engine/myvisualizer.py#L115

i believe this is the problem

[HopeFinder]

hi, @misraya ,have you dealt with this problem ? I meet with the same issue as you.