Closed ZhenYangIACAS closed 8 months ago
Well, one possibility is that your training loss has indeed undergone some steep change, so the high values in the past largely influence the global average. Another possibility is that your training data is composed of items covering a large range of difficulty. So a small proportion of items may have very large loss values that would influence the average loss, but due to the small proportion, they may not be reflected by the median value. There could also be other reasons.
Hi, I get very strange training loss: Averaged stats: lr: 0.000015 closs: 2.2352 (9.5134) grad_norm: 1.9342 (1.2562) Averaged stats: lr: 0.000030 closs: 0.2981 (9.2237) grad_norm: 0.5920 (1.0241)
I want to know why there is so much difference between the median loss and global averaged loss?