search

kohya-ss / sd-scripts

Apache License 2.0
5.31k stars 880 forks source link

Meaningless Loss in tensorboard. #1712

Closed iqddd closed 1 month ago

iqddd commented 1 month ago

image Why does the Loss value seem to have no meaning (neither current, nor averaged, nor averaged over an epoch)? It is set to some value at the very beginning of training (within the first epoch) and then fluctuates within a small range thereafter. Shouldn't Loss be measured at the end of an epoch on validation images?

kohya-ss commented 1 month ago

The average loss is the moving average of the losses of the steps of 1 epoch up to that point. Therefore, there will be large fluctuations at first, and then the fluctuations will become smaller.

We are planning to implement verification loss, so please wait for a while