How much loss should one expect at 200,000 iterations?

[sourabh-patil]

Hi team, I am training the model as per given instructions. At the starting point loss in the range of 10 power 5. Even after training the model for more than 200,000 iterations, the loss has decreased 10 times only (its in the range of 10 power 4). I am uploading 2 log files, kindly look at them. While training, deformable convolutions' offset seem to go above certain limit, but after some iterations it became normal (You can observe this in 2nd log at around166,000th iteration 1st.log 2nd.log

).

[Mukosame]

Hi @sourabh-patil , the loss should be always fluctuate a lot during the later training phase. Your loss looks fine for me. Besides, it is a dangerous alert when the offset is too high --- which means the model doesn't converge. It's good if it goes back to normal.

[sourabh-patil]

Thank you for your reply. Please can you tell me an approximate range of values of loss when the model finally converges (around 600,000 iterations right?!).

[Mukosame]

Hi Sourabh, it denpends on your batch size. The model actually converges every 150,000 iterations since we set periodic learning rate in the option file. Your range looks correct. Comparing to the training loss, it's better to tell if the model converges by testing on a validation set in every checkpoint. The PSNR values are more stable than the loss.

[sourabh-patil]

Oh I see. Thank you for the explanation.