search

facebookresearch / swav

PyTorch implementation of SwAV https//arxiv.org/abs/2006.09882
Other
1.99k stars 280 forks source link

loss behaviour #88

Open chinmay5 opened 2 years ago

chinmay5 commented 2 years ago

Thank you for the very nice work. I have a small doubt though. I am trying to train the model on a custom dataset and the loss decreases initially from ~8 to ~5 at the end of 60 epochs but then, it begins to increase again. I was wondering if someone else has faced a similar issue.

Thank you

ha-lins commented 2 years ago

Same issue @chinmay5 on my custom dataset. Have you addressed the problem of loss increasing? My loss decreased at the first some steps and increased later.

chinmay5 commented 2 years ago

Hi, @ha-lins I managed to do it by playing around with the crop-size and batch-size hyper-parameters. In short, it was more of exploring the hyper-parameters. Still, there was a specific scenario in which I could not manage to handle the issue. It may so be that your data clusters collapse (I got nan loss in that case).

zhangtzq commented 11 months ago

I got NAN loss after the 250 iterations on my dataset. Can you share with me how you tackle it, please?

GSusan commented 11 months ago

您好！我已收到邮件，会尽快回复。