search

stared / livelossplot

Live training loss plot in Jupyter Notebook for Keras, PyTorch and others
https://p.migdal.pl/livelossplot
MIT License
1.29k stars 143 forks source link

WARNING : tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time #131

Open essanhaji opened 3 years ago

essanhaji commented 3 years ago

tensorflow:Callback

Hi, Thank you very much for this amazing tool it helps a lot. Just a have a simple warning that will make this tool great if it is fixed. This warning occurs when I use PlotLossesKeras callback.

WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.1298s vs `on_train_batch_end` time: 0.2397s). Check your callbacks.

Environment

Thank you

Bartolo1024 commented 3 years ago

Do you use Livelossplot without any other callbacks? We don't have on_train_batch_end method in our callback (only 'on_epoch_end').

Noah-Junseo commented 2 years ago

Multiprocessing supported by TensorFlow itself is not supported on Windows. This is a suggestion to improve it by using the multiprocessing provided by Python directly. https://docs.python.org/en/3.7/library/multiprocessing.html

And, to see if multiprocessing is running, you need to look at what each process is doing, not the overall cpu stats.

In Linux, you can check by simply creating and disappearing multiple python processes with the same command as top, but I don't know Windows well, so if you google it, I think it might not come up.