search

dyelax / Adversarial_Video_Generation

A TensorFlow Implementation of "Deep Multi-Scale Video Prediction Beyond Mean Square Error" by Mathieu, Couprie & LeCun.
MIT License
734 stars 184 forks source link

volatile GPU-Util 0% #10

Open somuchsentimentality opened 7 years ago

somuchsentimentality commented 7 years ago

Hi dyelax,

I really appreciate for your work. I tried to modified your code to Wgan and trained on AWS. But it seems like training pretty slow, it only trained 4000 iterations a day. Also it has high GPU Memory-Usage but zero volatile cpu-util. Do you have any idea why does this happening?

(tensorflow) ubuntu@ip-172-31-8-95:~$ nvidia-smi Sun Apr 23 19:53:29 2017
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 375.51 Driver Version: 375.51 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla K80 Off | 0000:00:1E.0 Off | 0 | | N/A 40C P0 71W / 149W | 10941MiB / 11439MiB | 0% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 20379 C python 10937MiB | +-----------------------------------------------------------------------------+

Thank you!

dyelax commented 7 years ago

Sorry for the delay on this. Are you getting the same results if you try running this code without your WGAN edits?