search

sthalles / SimCLR

PyTorch implementation of SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
https://sthalles.github.io/simple-self-supervised-learning/
MIT License
2.21k stars 458 forks source link

Question about CE Loss #16

Closed WShijun1991 closed 4 years ago

WShijun1991 commented 4 years ago

Hello,

Thanks for sharing the code, nice implementation.

The way you calculate the loss by using a mask is quite brilliant. But I have a question.

logits = torch.cat((positives, negatives), dim=1) So if I'm not wrong, the first column of logits is positive and the rest are negatives.

labels = torch.zeros(2 * self.batch_size).to(self.device).long() But your labels are all zeros, which means no matter positive or negative, the similarity should low.

So I wonder is the first column of labels supposed to be 1 instead of 0.

Thanks for your help.

WShijun1991 commented 4 years ago

Sorry, I understand your code now

xuehaouwa commented 3 years ago

Sorry, I understand your code now

Hi, could you please help me understand it? I have the same question as you

WShijun1991 commented 3 years ago

@xuehaouwa They put cos_sim values of positive pairs as the first column of a matrix. And cos_sim of negative pairs are the rest columns of this matrix. Now in this matrix, you can consider each row logits, the 0th element is the "right" class, you can do CE on each row and expect the model to pick the right "class"(the positive pair). That is why they have a zero-value vector "labels".

321hallelujah commented 3 years ago

sorry, I still can not get it. What's "the 0th element"mean. And how to calculate 'loss' with a zero-value vector "labels". Can you explain more about it? grateful