search

alexlee-gk / video_prediction

Stochastic Adversarial Video Prediction
https://alexlee-gk.github.io/video_prediction/
MIT License
303 stars 65 forks source link

Training stability & progress #31

Open Cpruce opened 5 years ago

Cpruce commented 5 years ago

@alexlee-gk

Thanks for an awesome repo and architecture! I am currently training on my own dataset with batch size 8 and have been playing around with lower learning rates.

While the predicted frames are decent, my metrics seem to have a high variance, indicating the training process can be made better for this data.

Do you have any suggestions on how to improve training stability and/or results, besides further decreasing the learning rate? Here are some screenshots from the tensorboard graphs.

Screen Shot 2019-05-18 at 8 18 19 PM Screen Shot 2019-05-18 at 8 18 45 PM Screen Shot 2019-05-18 at 8 19 00 PM Screen Shot 2019-05-18 at 8 19 48 PM Screen Shot 2019-05-18 at 8 20 03 PM
eslambakr commented 5 years ago

I have the same question, as while training on my own data-set I got the same curves. I am wondering why the generator loss is increasing?!

Any updates regarding this issue?

Thanks in advance.