search

hendrycks / ss-ood

Self-Supervised Learning for OOD Detection (NeurIPS 2019)
MIT License
264 stars 31 forks source link

Signs of the two score components for OOD detection #24

Open j-cb opened 2 years ago

j-cb commented 2 years ago

Hi,

in the scoring formula on page 7 in the paper, shouldn't the KL divergence of the classifier prediction from uniform be small for OOD inputs, and the rotation CE be large on OOD since the rotation head has not been trained to predict the original rotation on OOD inputs? I.e. one of the terms should have a minus sign, right?

If I read it correctly, the code uses different signs for those terms:

https://github.com/hendrycks/ss-ood/blob/2a284be3f36de467ed3f35d8fa61bd9be7e261d6/multiclass_ood/test_auxiliary_ood.py#L182-L185

where KL is positive CE minus the constant entropy of U.

lygjwy commented 2 years ago

Hi,

in the scoring formula on page 7 in the paper, shouldn't the KL divergence of the classifier prediction from uniform be small for OOD inputs, and the rotation CE be large on OOD since the rotation head has not been trained to predict the original rotation on OOD inputs? I.e. one of the terms should have a minus sign, right?

If I read it correctly, the code uses different signs for those terms:

https://github.com/hendrycks/ss-ood/blob/2a284be3f36de467ed3f35d8fa61bd9be7e261d6/multiclass_ood/test_auxiliary_ood.py#L182-L185

where KL is positive CE minus the constant entropy of U.

I think the code is correct. For OOD inputs, the whole loss, i.e., kl_div(rot_one_hot, rot_smax) -1 * kl_div(class_uniform_dist, classification_smax) is bigger than ID inputs.

zjysteven commented 1 year ago

The code is correct while the formulation in the paper (Sec. 4.1 Method) is not.