search

NVlabs / imaginaire

NVIDIA's Deep Imagination Team's PyTorch Library
Other
3.99k stars 444 forks source link

mode collapse #43

Closed hugo-xie closed 2 years ago

hugo-xie commented 3 years ago

hi, when I trained with coco_funit, In the first few epochs, the results are normal, but mode collapse appears from the 59th epoch. Is this normal? Will it also appear during your training? epoch_00063_iteration_000094000

hugo-xie commented 3 years ago

any suggestion？

Edwardkim0 commented 3 years ago

i'm also curious about that...

mingyuliutw commented 3 years ago

Mode collapse is still an open problem in GAN training, especially dealing with challenging distribution.

Are you using our default configuration file and training the model using 8 GPUs? Just curious if this has something to do with the batch size.

Edwardkim0 commented 3 years ago

image i have 2 gpus, and costom dataset. but same problem... i will try batch_size 2 and then comment

mingyuliutw commented 3 years ago

Maybe lower the learning rate or increase the reconstruction loss?

Edwardkim0 commented 3 years ago

@mingyuliutw thanks for replying this question.. but still mode collapse happens... i try lr = 3e-5 ~ 1e-5 , reconstruction loss 0.15 ~ 0.3. image

i have used this code for funit. (https://github.com/NVlabs/FUNIT) it works well... but funit has unnatural artifact, so i need coco-funit...

Edwardkim0 commented 3 years ago

any suggestion?

Edwardkim0 commented 3 years ago

any suggestion?

I collect more data about 10k each category. then mode collapse not happend.