SteveJunGao
commented
11 months ago Hi @pallgeuer, thanks for the great questions!
I think the divergence might be because the GAN is unstable to train. There're several opinions you can try to increase stability:
- Increase the batch size
- Turn off the fp16 training by doing
--fp32=1(we recommend to use--fp32 0only when you want the speed) - The most effective regularization is the R1 regularization from the StyleGAN, you can increase the
--gammawhen you see the instability (e.g. make it 2x larger) - We also observe the model can sometimes diverge during training, you can just restart the training and it will often disappear.
Re volume subdivision: Unfortunately, we didn't include the volume subdivision code in this codebase, our paper has the ablation studies when removing the volume subdivision (Table 2) and the released pre-trained model is trained without volume subdivision.
Re R1 regularization: the gamma we used is 80 for motorbikes.
Re SDF regularization: which regularization do you mean exactly? we do have one regularization in the paper (Eq 2) and the hyperparameter for this is fixed in the code for all the experiments
Re single v.s. two discriminators: we always use two discriminators in all the experiments (except when we do ablation studies to compare the effect of two discriminators v.s. a single discriminator)
Re Adam beta. We apologize for the typo in the paper, the (0, 0.99) in the code is the correct one.
pallgeuer
jingyang2017
Hi, I am trying to use this code to train on the motorcycle data but the training is proving to be unstable. I have done the blender renders as described and have all 337 models with 96 renders per model. I train as follows:
This should be essentially the default documented training parameters except that I'm running on 2xA100 instead of 8xA100.
My issue is that the FID50k only decreases from ~250 initially to ~85 (more than the 50-65 expected from the paper), and at around 2000-3000kimg (out of the planned 20000kimg) the training diverges and never recovers. What parameters should I use so that your code on your data can actually finish training?
I would also be interested to know what the differences are between the code and training commands provided in this github, and the one that was used to train the pretrained motorcycle model. For one, volume subdivision isn't implemented, but what else (e.g. R1 regularization, SDF regularization, single vs two discriminators)? The paper also says Adam beta = 0.9, but the code uses (0, 0.99) (!) which is puzzling.