facebookresearch / odin

A simple and effective method for detecting out-of-distribution images in neural networks.
Other
526 stars 102 forks source link

About computing FPR at 95% TPR used in calMetric.py #21

Open W-XuWX opened 1 year ago

W-XuWX commented 1 year ago

Hi, thank you so much for your work. I would like to understand better on your tpr95 method.

Q1. For obtaining the results of FPR at 95% TPR, I would like to understand why the values of start and end were different when comparing your algorithm vs the baseline method, which uses an end of 0.1 and 0.01 for CIFAR10 and CIFAR100 respectively?

def tpr95(name):
...
    # calculate baseline
    if name == "CIFAR-10": 
    start = 0.1
    end = 1 
    if name == "CIFAR-100": 
    start = 0.01
    end = 1    
...
    # calculate our algorithm
...
    if name == "CIFAR-10": 
    start = 0.1
    end = 0.12 
    if name == "CIFAR-100": 
    start = 0.01
    end = 0.0104    

Q2. For the same function, how are the values "if tpr <= 0.9505 and tpr >= 0.9495:" chosen for 95% TPR? As in is it based on 4 decimal places or some other approach. I experimented with other datasets and models, and found that there were zerodivision errors errors if the tpr was slightly above 0.9505 (like 0.9545 for example)

Thank you so much!