search

thuml / Self-Tuning

Code release for "Self-Tuning for Data-Efficient Deep Learning" (ICML 2021)
109 stars 14 forks source link

PGC_labels is constants ? #3

Closed feng-1985 closed 2 years ago

feng-1985 commented 2 years ago

code snippet of models.py( line 121-122): PGC_labels = torch.zeros([batch_size, 1 + self.queue_size*self.class_num]).cuda() PGC_labels[:,0:self.queue_size+1].fill_(1.0/(self.queue_size+1))

wxm17 commented 2 years ago

Yes. For any query, we put the positive samples at the first self.queue_size+1 positions, while the negative samples on the rest *self.queue_size (self.class_num -1)** ones. Since positive and negative samples have constant values, PGC_labels are constants.

feng-1985 commented 2 years ago

Thanks!

feng-1985 commented 2 years ago

@wxm17 I am testing the code, and found that the PGC_label_labeled is identical to PGC_label_unlabeled.

labeled tensor([0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303], device='cuda:0') unlabeled tensor([0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303, 0.0303], device='cuda:0')

Somthing is wrong ?

wxm17 commented 2 years ago

No, they are right. The PGC_labels here are used for contrastive learning, instead of the labels for classification. Therefore, the labels here are the indicators of positive samples and negative samples. PGC_label_labeled should be identical to PGC_label_unlabeled. More details can be found at https://github.com/facebookresearch/moco/blob/master/moco/builder.py

feng-1985 commented 2 years ago

Thanks for response! I just found the main idea is come from moco (Never heared of it, i am not the cv learner) and dig it a lot. Beside, I am trying to reproduce the result with name entity recognition task. Looking forward this part of code sharing.