search

stared / livelossplot

Live training loss plot in Jupyter Notebook for Keras, PyTorch and others
https://p.migdal.pl/livelossplot
MIT License
1.29k stars 142 forks source link

CPU version of Tensorflow training is very slow with PlotLossesCallback #52

Closed Kuselokusi closed 5 years ago

Kuselokusi commented 5 years ago

I have the CPU version of tensorflow and have been training a 3 layered Simple Recurrent keras model. However, the model takes takes too long to complete the an epoch (in excess of 30 minutes).

This started when I added the plot_losses = PlotLossesCallback() so that I can get live progress plots of the progress on the losses and accuracies for each epoch. I'm not sure how to remediate the issue.

Would installing cuDNN be helpful to fixing the issue or does it not work with the CPU version of tensorflow? Or is there another when this can be fixed?

stared commented 5 years ago

@Kuselokusi Could you compare training time with and without livelossplot?

In a notebook, use %%time.

Kuselokusi commented 5 years ago

Sorry I just figured out what the issue is and it is related to training time but more to the use of the livelossplot library.

More specifically, after each epoch a plot of the accuracies and losses on (validation and training sets) are displayed. But the issue is that after each epoch, the plot is displayed and I have to manually close it so that the training process continues. Then at the end of the next epoch the updated plots are displayed and I have to do it all over again.

Note that I am running my code using a batch file (that uses the command line), not in spyder or something similar. Please assist.

stared commented 5 years ago

I close this issue as non-conclusive (you didn't answer my question).

Here we deal with livelossplot, NOT issues related to deep learning frameworks.

...

From your last question, it seems that you run in NOT in Jupyter Notebook. For the command line, only Neptune interface is supported (see README).