Closed thesofakillers closed 1 year ago
Note: zooming into the plot reveals that the logging is indeed occurring every 50 steps during the first part of training, but simply not plotted when zoomed out for some reason. Seems to be a wandb bug. Nevertheless, weird that it gets triggered when min_epochs and early stopping conflict.
Before zooming in on 2 identical runs (brown one has not had the conflict occur yet)
After zooming in:
They are indeed identical.
Closing this and opening in wandb repo. Apologies.
Bug description
This morning I woke up to a very weird result.
I have a PL module/trainer/dm setup that
When i started training last night, everything was working fine, my training metrics were being logged every 50 steps and loss was decreasing.
This morning I woke up to see that
The high frequency logging you see at the end of training should be observed everywhere and was observed last night at the beginning of training before early stopping was triggered.
What version are you seeing the problem on?
v1.9
How to reproduce the bug
I have not been able to reproduce this bug with a smaller dataset/setup. Advice with how to try to reproduce it is welcome.
Environment
#1
SMP Thu May 11 07:38:47 EDT 2023More info
Here is my setup:
In my PL LightningModule, I call
self.log('visual/train_loss', loss, batch_size=batch_size)
in thetraining_step
, and similarly self.log('visual/train_loss', loss, batch_size=batch_size)in the
validation_step`. Not really sure what I am doing wrong. I ran the same exact run (same seed and everything) and verified that the logger indeed logs every 50 steps at the beginning, indicating that these indeed do get dropped once EarlyStopping gets triggered.