search

jayleicn / singularity

[ACL 2023] Official PyTorch code for Singularity model in "Revealing Single Frame Bias for Video-and-Language Learning"
https://arxiv.org/abs/2206.03428
MIT License
129 stars 13 forks source link

Why add 1e-4 in softmax #27

Closed Andy1621 closed 1 year ago

Andy1621 commented 1 year ago

Hi! I found that 1e-4 is added in the softmax function. Are there any reasons? https://github.com/jayleicn/singularity/blob/bf4a86ec7506565d1f6805ee1612aa6029592776/models/model_retrieval_base.py#L303-L304

If it is used for smoothing, why not add it after softmax, like F.softmax(sim_i2t, dim=1)+1e-4.

jayleicn commented 1 year ago

To avoid zero-division error in F.softmax.

Andy1621 commented 1 year ago
image

The softmax uses exp. It seems that it will not cause zero-division error.

jayleicn commented 1 year ago

Oh, you are right! I was looking at it too quickly. Please see this issue https://github.com/salesforce/ALBEF/issues/93

Andy1621 commented 1 year ago

Thanks!