ghost
commented
6 years ago Sorry to bother you again, but I switched to a Tesla V100 and it seems to have stalled out around 40% "real accuracy". It was progressing smoothly but now it's crawling. Training on a K80 also kind of stalled here too so maybe that's normal, or maybe something's off. I don't know though...
The GANgogh.py architecture and other files are unchanged except for a few edits to make it run in the cloud. Here's my average output for the last hour or two: 2017-12-01 20:27:18,077 INFO - iter 14599 train disc cost -0.01166571956127882 time 0.5838999676704407 wgan train disc cost -1.9136346578598022 train class cost 1.7541061639785767 generated class cost 0.5280181169509888 gen cost cost 413.5545349121094 gen accuracy 0.9763096570968628 real accuracy 0.4273809790611267
Sometimes GANs have a difficult time converging, so does the 0.97 gen accuracy and 0.42 real accuracy mean that the generator maximized it's reward and overpowered the discriminator, or that the discriminator is guessing 50/50 +-10 and we're reaching convergence? Still kind of new to GANs so any help or suggestions would be really appreciated :)
Update: it eventually crawled it's way to ~49% real accuracy and ~98% gen accuracy. Is that what we were aiming for?
How long did training take originally and on what kind of GPU? I've got it running on a K80, but I pay by the hour so it would be great to have a vague idea of how long I should wait to see moderate results and/or if I need to adjust something before wasting more compute hours.