How to search the unknown threshold from train set

[LinyeLi60]

Hi wentao, I can not figure out where is the code to get the unknown threshold of equation 11 in paper. I find visualization.py and search_param.py, so which is the right one ? Thank you!

[Cogito2012]

Hi Linye, the visualization.py is the correct one, and the search_param.py is actually not used.

Note that the threshold in equation 11 is only used for inference/visualization purposes, which can be found in visualization.py (Line113). For quantitative evaluation, we do not need threshold.

[LinyeLi60]

the threshold in equation 11 is only used for inference/visualization purposes so the threshold in equation 11 will not influnce the evaluation results? In my opinion, the threshold is a very important hyper-parameter because it will decide a sample is 'unknown' or 'known' class, which will influnce the evaluation results greatly. Forgive me for asking such a levelless question.

[Cogito2012]

No, threshold will not be used for evaluation metrics such as AUROC, AUPR, and OSDR, because these metrics calculate the area under a certain curves, which are determined by varying thresholds from 0 to 1.

However, threshold definitely will impact the results using other metrics like micro- or macro-F1, precision, recall, etc. Therefore, to get rid of the impact from threshold values, we encourage the whole community to discard those threshold-dependent metrics for OSR/OOD tasks.

If you still want to get the threshold using training dataset, you could refer to the AFSD/thumos14/threshold.py.

[LinyeLi60]

Thank you! My work is about open set object detection, so I need to get the threshold to get better mAP results, is it right?

[Cogito2012]

Right, mAP is also impacted by threshold. But please remember that only training set can be used to get threshold for mAP calculation.

[LinyeLi60]

Ok. Thank you for your patient guidance !!!

Closed.