search

sthalles / SimCLR

PyTorch implementation of SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
https://sthalles.github.io/simple-self-supervised-learning/
MIT License
2.21k stars 458 forks source link

'CosineAnnealingLR' never works with the wrong position of 'scheduler.step()' #8

Closed GuohongLi closed 3 years ago

GuohongLi commented 4 years ago

Considering the setting in 'scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=len(train_loader), eta_min=0, lastepoch=-1)'，I think 'scheduler.step()' should be called every step in 'for (xis,xjs), in train_loader'. Otherwise，lr will nerver change until 'len(train_loader)' epochs but not steps

GuohongLi commented 4 years ago

Mistake Fixs: "lr will nerver change"->"lr will nerver change over multiple cosine periods"

sroutray commented 4 years ago

I think in the paper they mention that they follow cosine learning rate decay "without restarts". I interpreted it as decaying the learning rate after 10 epochs (warm start) and decay it over epochs to the minimum value. Given this, the T_max arguement should be equal to the number of epochs. Check this too: https://github.com/Spijkervet/SimCLR/blob/2b900bb4972b5c8792e9bdfc138875beb626a19c/model.py#L6

sthalles commented 3 years ago

Hello guys, that is a general message to say that I have refactored the whole project. I believe the project is much easier to understand now. Please have a look at the new impl and free to submit PR if you find any bugs. Thanks.