search

zhihou7 / BatchFormer

CVPR2022, BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning, https://arxiv.org/abs/2203.01522
242 stars 20 forks source link

The loss of BatchFormer with MoCo-v3 ? #18

Open iorange-77 opened 1 year ago

iorange-77 commented 1 year ago

May I ask the 133-134 lines of code in ./moco-v3/moco/builder.py: loss = self.contrastive_loss(q1[:N], k2[:N]) + self.contrastive_loss(q1[:N], k2[N:]) + self.contrastive_loss(q1[N:], k2[:N]) + self.contrastive_loss(q1[N:], k2[:N]) + \ self.contrastive_loss(q2[:N], k1[:N]) + self.contrastive_loss(q2[:N], k1[N:]) + self.contrastive_loss(q2[:N], k1[:N]) + self.contrastive_loss(q2[:N], k1[N:])

Is the loss calculated incorrectly? should be: ？？ loss = self.contrastive_loss(q1[:N], k2[:N]) + self.contrastive_loss(q1[:N], k2[N:]) + self.contrastive_loss(q1[N:], k2[:N]) + self.contrastive_loss(q1[N:], k2[N:]) + \ self.contrastive_loss(q2[:N], k1[:N]) + self.contrastive_loss(q2[:N], k1[N:]) + self.contrastive_loss(q2[N:], k1[:N]) + self.contrastive_loss(q2[N:], k1[N:])

If my understanding is wrong, what does this loss calculation method mean?

zhihou7 commented 1 year ago

Dear @iorange-1014, Many Thanks to you! It is a typo. It should be self.contrastive_loss(q2[N:], k1[:N]) + self.contrastive_loss(q2[N:], k1[N:]) in the last! I have updated the code.

Sincerely yours,

zhihou7 commented 1 year ago

Hi @iorange-1014, Thanks for your comment. Sorry for the terrible code. It seems like the model achieves clearly better performance with the revised code. The pretrained model is provided in here. Actually, I still think current losses might include redundant terms. I am not familiar with moco. And I do not have enough computational resources to tune this, but providing a naive and intuitive version. Lastly, many thanks to you.

Regards,