Reproducing OOD detection results

[swyoon]

Dear Du,

First of all, thank you for your awesome work. I have been always a fan of your fantastic line of work on energy-based models.

I am reaching out because I am having some trouble reproducing the out-of-distribution (OOD) detection results presented in the Improved Contrastive Divergence paper. Specifically, I am getting weaker results than what was reported in the paper.

I ran the same command provided in the readme file, using the following settings:

python train.py --exp=cifar10_model --step_lr=100.0 --num_steps=40 --cuda --ensembles=1 --kl_coeff=1.0 --kl=True --multiscale --self_attn

The training ran for over a day on V100, and Tensorboard reports that the inception score is higher than 5, which leads me to believe that the training process was successful.

However, when I attempted to perform OOD detection using the best checkpoint and other checkpoints, I got AUC scores between 0.72 and 0.78. This is lower than the AUC scores presented in the paper.

I was wondering if you could offer any advice on how to reproduce the AUC scores reported in the paper, or if you have any tips for achieving good OOD detection performance.

Thank you for your time and help.

Best, Sangwoong.

[yilundu]

Hi Sangwoong,

Thanks for your interest in your paper -- its been a while since I ran the code but I believe I got the AUC score using not the best trained checkpoint but another checkpoint that had high OOD score. To obtain improved OOD detection performance I found that using different amounts data augmentations from when sampling from the replay buffer to be quite helpful.

Best, Yilun

[swyoon]

Thank you so much for such a quick reply. Your answer helped a lot.

Btw, congratulations on your new paper, Reduce, Reuse, Recycle. I also enjoyed reading the paper as well :)

Have a good day.

Best, Sangwoong.