I noticed when I trained my network from scratch. First epoch always ended up with very high loss and classification_loss. e.g 200 ish.
It make partially sense because this is just the start. However, when second epoch starts, I saw loss drops to 1 or 2. Still high but way less than the first epoch. from 200 to 2.
My understanding is that each epoch should pick up where you left from last epoch, so loss should be consistent. It is not a big issue though, as it won't hurt consecutive epochs. But I still wondering why such weird case happened.
I make some changes to resnet50 backbone(add more layer etc.) And my training arg does not include --imagenet-weight and --freeze-backbone. I know imagenet weight will be initialized by default, but it doesn't help to explain loss inconsistant. Do second epoch not follow the weights from the first and it just restart all over?
Hi,
I noticed when I trained my network from scratch. First epoch always ended up with very high loss and classification_loss. e.g 200 ish.
It make partially sense because this is just the start. However, when second epoch starts, I saw loss drops to 1 or 2. Still high but way less than the first epoch. from 200 to 2.
My understanding is that each epoch should pick up where you left from last epoch, so loss should be consistent. It is not a big issue though, as it won't hurt consecutive epochs. But I still wondering why such weird case happened.
I make some changes to resnet50 backbone(add more layer etc.) And my training arg does not include --imagenet-weight and --freeze-backbone. I know imagenet weight will be initialized by default, but it doesn't help to explain loss inconsistant. Do second epoch not follow the weights from the first and it just restart all over?
Thanks in advance.