search

pokaxpoka / deep_Mahalanobis_detector

Code for the paper "A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks".
340 stars 78 forks source link

AUROC with M(x) scores? #10

Closed flpgrz closed 2 years ago

flpgrz commented 2 years ago

While reading the paper I struggle to understand the following:

how to compute a AUROC score using the M(x) distance score? If the ground truth is 1, for in-distribution, and 0, for out-of-distribution, how to compute a AUROC if M(x) is e.g. - 639.2 (i.e. not a distribution)?

Thanks for your help!

ForeverPs commented 2 years ago

For example, you have the target labels (0 or 1) of testing data, corresponding, you can get the M(x) of each testing data. With all predicted results and the target labels, you can calculate the AUROC, AUPR_in AUPR_out and some other metrics.

flpgrz commented 2 years ago

Hello @ForeverPs, thanks for the reply.

Yes, I agree with you. What was confusing me is that M(x) is not a probability. I was convinced that computing the AUROC requires a probability, i.e. a score in [0,1]. But actually I observed that you can compute AUROC also with any kind of predicted score.

ForeverPs commented 2 years ago

Hello @ForeverPs, thanks for the reply.

Yes, I agree with you. What was confusing me is that M(x) is not a probability. I was convinced that computing the AUROC requires a probability, i.e. a score in [0,1]. But actually I observed that you can compute AUROC also with any kind of predicted score.

Yes, you are right. This metric is based on sorting.