search

vturrisi / solo-learn

solo-learn: a library of self-supervised methods for visual representation learning powered by Pytorch Lightning
MIT License
1.41k stars 182 forks source link

Cosine learning rate schedule is weird in the updated version June 03 #248

Closed trungpx closed 2 years ago

trungpx commented 2 years ago

Hello,

I used this library before. Today I just clone the latest one from this website. But when running the baseline of BYOL, I found its tau seems to be incorrect as picture below. Am I correct or this is just an intended update from you, Vturrisi?

Figure1. It looks not correct since previously it increases from 0.99 to one as in Fig. 2 I ran before. image

Below I looked up again the previous runs. Figure 2. A reference to BYOL ran 1 month ago. image

DonkeyShot21 commented 2 years ago

Hi, we fixed the issue yesterday. Can you pull and try again?

trungpx commented 2 years ago

I'm trying and will get back soon.

trungpx commented 2 years ago

Thanks for the fixing. It's correct now. Due to this issue, all other baselines using momentum I ran 2 days ago are wrong. I'm running again all of them with the fixed one.

To summarize issue and code fixed, the change in base.py as follows:

Yesterdays (wrong): self.momentum_updater.update_tau(cur_step=cur_step, max_steps=self.num_training_steps) Today (fixed --> worked): self.momentum_updater.update_tau(cur_step=cur_step, max_steps=self.max_epochs * self.num_training_steps)