search

EugenHotaj / pytorch-generative

Easy generative modeling in PyTorch.
MIT License
422 stars 68 forks source link

How to view progress during training? #28

Closed AlexanderGroeger closed 2 years ago

AlexanderGroeger commented 3 years ago

I've ran a PixelCNN in Colab last week, but I didn't know when it would finish (4 hours) because there was no in-training output. If you're concerned about flooding a cell with outputs, you could use tqdm over all the epochs that way the user can still see progress while printing only a few lines.

EugenHotaj commented 3 years ago

@AlexanderGroeger the Trainer writes out TensorBoard logs to the logdir directory, e.g:

Screen Shot 2021-05-18 at 9 39 06 AM

You can spin up a TensorBoard server if you're running locally with the command tensorboard --logdir <logdir>, or if you're running on Colab with the magic %tensorboard --logdir <logdir>. Does that work for you?

We used to have tqdm but it wasn't that helpful because the output becomes overwhelming when you have a lot of metrics and it's easier to interpret metrics in a chart vs. raw dumps. Maybe having a simple bar like you suggest would be useful though. I'm happy to accept a pull request if you want to add this.

AlexanderGroeger commented 3 years ago

I'm still new to tensorboard, so I'll see if I can view it live. Thanks.