Currently we wait for training to complete and then plot the loss versus epoch, and diagnoses if the model parameters seem reasonable. Sometimes the training can go off the rails, such as with the profile coefficients. It is helpful to diagnose these problems in real-time so we can pause training and fix parameters or make some change without having to wait for training to complete.
Tensorboard's add_scalar method will allow us to see real-time updates of the training progress.
Currently we wait for training to complete and then plot the loss versus epoch, and diagnoses if the model parameters seem reasonable. Sometimes the training can go off the rails, such as with the profile coefficients. It is helpful to diagnose these problems in real-time so we can pause training and fix parameters or make some change without having to wait for training to complete.
Tensorboard's
add_scalarmethod will allow us to see real-time updates of the training progress.