search

HobbitLong / CMC

[arXiv 2019] "Contrastive Multiview Coding", also contains implementations for MoCo and InstDis
BSD 2-Clause "Simplified" License
1.3k stars 179 forks source link

Implementing CMC on CIFAR-10 #50

Closed ShaoTengLiu closed 4 years ago

ShaoTengLiu commented 4 years ago

Hi @HobbitLong, I am trying to implement CMC on CIFAR-10 with a shallow ResNet. However, the accuracy only reaches 60%~70%. I have tried to tune the batch size from 64 to 512 and learning rate from 0.01 to 0.12. In addition, I also tuned the nce_k from 8192 to 65536. Unfortunately, it is not improved yet. I am writing to ask do you have any suggestions on tuning parameters on small datasets like CIFAR-10? Thank you very much.

ShaoTengLiu commented 4 years ago

Update. I think this may be caused by #27. I find there seems no method to prevent an element in the enqueue come from the same sample as the query. As a result, the loss increases a lot at the beginning of each epoch.

截屏2020-03-24下午2 33 51

This may not hurt performance when the dataset is large enough. However, for CIFAR-10, which only consists of 50,000 samples, this may hurt the performance a lot.

ShaoTengLiu commented 4 years ago

Update. I find you gave some hint on this problem, as shown in hint. Can I conclude that we should use softmax and less negative samples to implement CMC on small datasets?

HobbitLong commented 4 years ago

@ShaoTengLiu ,

Sorry for the late reply, I have obtained ~70% on CIFAR100. I think using 2048 might suffice on CIFAR datasets (10 or 100). Also you could sample without replacement to solve the conflict issue you just pointed to.

ShaoTengLiu commented 4 years ago

@HobbitLong ,

Thanks for your reply. Yes, 2048 or 4096 is suffice on CIFAR. The problem has been solved. You can close this issue. Thank you very much.

HobbitLong commented 4 years ago

Glad that you have figured it out. Feel free to reopen for future issues!

WeihongM commented 4 years ago

@HobbitLong ,

Thanks for your reply. Yes, 2048 or 4096 is suffice on CIFAR. The problem has been solved. You can close this issue. Thank you very much.

Hello， @ShaoTengLiu Can you share the performance you reach when using cifar-10 dataset.