search

HobbitLong / SupContrast

PyTorch implementation of "Supervised Contrastive Learning" (and SimCLR incidentally)
BSD 2-Clause "Simplified" License
2.98k stars 525 forks source link

CE vs CE + SupCon #136

Closed qsunyuan closed 10 months ago

qsunyuan commented 10 months ago

Great job! SupCon is very inspiring to me.

I have some questions and thoughts that I would like to ask the author. Since contrastive loss is used during the pre-training stage to learn better representations, followed by measuring embedding quality using a Linear Classifier in downstream tasks. I'm curious if there have been any results from training the model from scratch directly in downstream task (e.g. directly training CIFAR-100 classification task), simultaneously using both SupCon and Cross-Entropy losses? I would like to know if this approach yields better results compared to a model trained solely with Cross-Entropy loss.

HobbitLong commented 10 months ago

Thank you for the kind words!

I haven't run experiments w/ "SupCon + CE" setup on CIFAR-100. Many training techniques has been proposed to improve the vanilla CE training (e.g., this one), since the publishing of SupCon. Some of those may potentially help SupCon too. So I would recommend running some experiments to check.

jjwei66 commented 7 months ago

@qsunyuan Excuse me, have you tried the 'SupCon + CE' setup? I also wonder if it will bring better results.