search

salesforce / PCL

PyTorch code for "Prototypical Contrastive Learning of Unsupervised Representations"
MIT License
570 stars 83 forks source link

are something wrong in the builder of PCL? #5

Closed ZhiyuanDang closed 3 years ago

ZhiyuanDang commented 4 years ago

https://github.com/salesforce/PCL/blob/5c8b722783cf130d17a1e6bb26713dcb3545d6d0/pcl/builder.py#L189

After carefully reviewing the code, I think the torch.linspace in the above line should be torch.zeros since the positive prototypes have been concatenated the first dim.

LiJunnan1992 commented 3 years ago

Hi, thanks for your interest!

pos_prototypes have a size of B x low_dim, thus in logits_proto, for the first B x B elements, the diagonals correspond to positive logits. note that each prototype in pos_prototypes are considered negative prototypes for other samples in the mini-batch.

codergan commented 3 years ago

have a size of B x low_dim

may I ask, if two data points in same mini-batch belonged to same pos_prototypes, they will still be considered as negative prototypes?

LiJunnan1992 commented 3 years ago

Hi, the negatives are selected in a way that exclude all positives in the mini-batch.

codergan commented 3 years ago

Hi, the negatives are selected in a way that exclude all positives in the mini-batch.

Yes, I mean for two data points in one mini-batch, if they belong to same prototype, in the loss calculation for one data point, the other will be considered as negative prototype right? since the prototype label is torch.linspace(0, q.size(0)-1, steps=q.size(0)).long().cuda() , which only consider itself.

LiJunnan1992 commented 3 years ago

Yes you are right. The chance of this happening is very low, though, given the large number of prototypes compared to batch size. Thanks!