Open dvkeulen opened 4 years ago
When the checkpoint is saved while training, how long should this process take? It seems to me that my program doesn't run at the point of saving.
In the image I set the checkpoint saving threshold to 10 for testing purposes.
edit: it does finish it, but it takes 2 whole hours to save the checkpoints. Is this normal behaviour?
That is preparing the test data.
When the checkpoint is saved while training, how long should this process take? It seems to me that my program doesn't run at the point of saving.
In the image I set the checkpoint saving threshold to 10 for testing purposes.