ClashLuke
commented
4 years ago Doing some exploration using the DCGAN Tutorial as a base, I was able to find out that 1.1) Hinge Loss makes the d_error drop to 0 if no penalty is used 1.2) If consistency regularization is used (even at gamma=10), the discriminator converges significantly slower than it does with BCEWithLogitsLoss 2) After 1000 updates using RootTanh plugged in to the default DCGAN, the generator already is stronger than it would be after 7000 updates as a ResNet (residual path=scale, module path=5x5/4x4 followed by 3x3 (performs better than 1x1))
EDIT: It's the penalty. It converges differently, without any noticeable speed difference, but significantly more stable. However when adding the penalty, the convergence slows down a lot.
Exploration
The element of a group giving the best results is marked in bold. If multiple elements are used together, all of them are marked. \ If a minor positive changes were noted in the early training, an item is marked using cursive text. However, it doesn't mean that it's competitor network can't overcome those slight disadvantages.\ As it's a GAN, absolute performance can not be measured using accuracy and FID and other metrics would be a waste of time for GAN-breaking changes. Therefore a scale from breaking (--) to stabilizing/drastically improving performance (++) is created, with 0 symbolizing indifference. Note that a breaking change blocks the model from converging any further than generating blobs of the wrong colour, but does not necessarily lead to pure noise.\ All is compared to the baseline DCGAN.