Spikes in training loss

google-research / simclr

SimCLRv2 - Big Self-Supervised Models are Strong Semi-Supervised Learners

https://arxiv.org/abs/2006.10029

Apache License 2.0

4k stars 621 forks source link

Spikes in training loss #189

Open slala2121 opened 2 years ago

slala2121 commented 2 years ago

I'm wondering if you have encountered this recurring spike in the training loss and why might this occur.

It does appear to occur upon processing the last batch but I don't think it has to do with uneven batch size because the last batch is dropped. The data is shuffled so it can't be any particular examples causing this. But it's not clear why the processing of the last batch would produce these spikes.

Thanks.

Screen Shot 2022-02-06 at 12 37 55 AM

thecooltechguy commented 1 year ago

@slala2121 I'm actually experiencing the exact same behavior, and I'm also shuffling + dropping the last batch. Did you happen to figure out the reason or resolve this?