Reproducing classification results with WRN

[Becktor]

Hi,

First of thanks for this great work.

I am currently struggling to reproduce the WRN model you have provided, with a model I train myself. (Training on CIFAR10)

With your provided model I reach an FPR95 of roughly 24, but using the provided hyperparameters I only reach an FPR95 at around 27.

Have I missed something or are there more hyperparameters to tune other than: --start_epoch 40 --sample_number 1000 --sample_from 10000 --select 1 --loss_weight 0.1

Also, I was wondering why you use the FPR95 differently than the FPR95 proposed by Deep Anomaly Detection with Outlier Exposure as your classification codebase is modified from them. (Not using the OOD samples as the positive class)

[d12306]

hi, @Becktor , thanks for the interest in our work!

I am not sure about your results, but I can reproduce the similar results using the current hyperparameters. Could you please launch multiple runs to see the results? Also, the FPR is more sensitive to randomness, if the AUROC is close, then it should be fine.

For the calculation of FPR, we follow the definition and code of https://github.com/wetliu/energy_ood.

[Becktor]

Thanks for the reply.

Fair enough, I'm able to reach a FPR of 24 if I follow the definition that OOD samples (the way described in the paper refrenced above) are seen as the positive class but not otherwise. (I might be misunderstanding the way you use it, correct me if I'm wrong but you follow the case where IID samples are used as positive class)

Anyways it might be due to different hardware. Currently sampling from the ood datasets 10 times and the std is pretty small so it shouldn't make that big a difference.

I am fairly close to the results presented so I was just checking if I had misunderstood anything drastic.