search

liyunsheng13 / BDL

MIT License
222 stars 30 forks source link

How do you decide when to stop the training ? #12

Open idealwei opened 5 years ago

idealwei commented 5 years ago

Hi, I am curious how to decide when to stop the training and how to choose the final snapshots. It's not clarified in your paper. I found the "Early Stopping" parameters in your code, how to set this hyper-parameter?

liyunsheng13 commented 5 years ago

It is a little hard to decide. For training without using SSL, I find stopping at 80000 iterations is best. When I continue to train with more iterations, overfitting will be caused. For training with SSL, I find there is not overfitting, but after 120000 iterations the mIoU starts to be stable. So I choose to stop at 120000 iterations for SSL.

idealwei commented 5 years ago

In my opinion, when training without SSL, you tested every snapshot on 'val' split of Cityscapes and set the best snapshot iteration as the "Early Stopping" hyper-parameter.

liyunsheng13 commented 5 years ago

Not really. The best result is always shown when iteration is around 80000. It is unnecessary to validate all snapshots.

idealwei commented 5 years ago

But if you have not got the performance of 75000 , 85000, 90000, how did you know that the best results is shown around 80000.