search

Jingkang50 / OpenOOD

Benchmarking Generalized Out-of-Distribution Detection
MIT License
831 stars 106 forks source link

Implementation of ASH seems incorrect. #233

Closed mgranz closed 5 months ago

mgranz commented 5 months ago

While going through the implementation of ASH, I noticed that the thresholding is not done input-dependent.

ASH zeros out a percentile of the lowest activations of each sample. In your implementation, it seems that the calculation of the threshold that zeros out the lowest percentile is missing. Instead, only self.percentile is given as a threshold, which cuts the activations at a threshold between [0,1].

I hope I am not missing something in your code and understood it correctly. If so, I hope that I was able to help identify this bug.

zjysteven commented 5 months ago

I could see where this confusion comes from. When running ASH, the ASH-specific model wrapper will be applied, https://github.com/Jingkang50/OpenOOD/blob/18c6f5174a2f518e2a8e819ffb1cd1914bcf12e0/openood/evaluation_api/evaluator.py#L115-L116 where the forward_threshold will call corresponding ASH processing function to compute the threshold according to percentile. https://github.com/Jingkang50/OpenOOD/blob/18c6f5174a2f518e2a8e819ffb1cd1914bcf12e0/openood/networks/ash_net.py#L17-L22 You are probably looking at vanilla model's forward_threshold, which is used for ReAct.

zjysteven commented 5 months ago

Closing now. Feel free to reopen if you have more questions.

mgranz commented 5 months ago

Thank you for the quick response! I indeed thought that forward_threshold, e.g. from resnet18_32x32 class, would be called. My bad!