search

POSTECH-CVLab / PyTorch-StudioGAN

StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.
https://github.com/MINGUKKANG
Other
3.42k stars 342 forks source link

ReACGAN + ADC has lower performance than plain ReACGAN #184

Open Adversarian opened 1 year ago

Adversarian commented 1 year ago

Hi, first of all I would like to thank you tremendously for your work. I'm certain that this repository alone has saved countless hours for researchers and developers all across the world.

I am conducting a series of experiments on image generation using ADCGAN and ReACGAN and I just noticed that, against my expectations, ReACGAN + ADC actually has a worse performance than plain ReACGAN on CIFAR100, resulting in a lower Inception Score and a higher FID.

I'm using the default configuration files provided within the repo, namely ReACGAN-ADC-DiffAug.yaml and ReACGAN-DiffAug.yaml. I also conduced the training procedure twice at 200k steps each to make sure I haven't hit an odd outlier the first time.

I would like to inquire as to why this could be happing since it appears counterintuitive that adding an ADC component to ReACGAN should be negatively affecting its performance, as it doesn't seem there should be any apparent downsides to this augmentation from a theoretical standpoint. I would appreciate your insight on the matter.

Thank you in advance and please do let me know if you need any further information.

mingukkang commented 1 year ago

I am conducting a series of experiments on image generation using ADCGAN and ReACGAN and I just noticed that, against my expectations, ReACGAN + ADC actually has a worse performance than plain ReACGAN on CIFAR100, resulting in a lower Inception Score and a higher FID.

We have encountered the same issue, and I am glad that you brought it up.

How about the result on CIFAR10? My assumption is that the increased number of classes may have contributed to the training instability, given that training ReACGAN with ADC trick is generally more challenging than vanilla ReACGAN training. Despite this, I still believe that there should be no theoretical issues. Thus, I am interested in exploring why the results did not meet our expectations:)

Adversarian commented 1 year ago

Due to certain constraints on my training configuration, I was not able to evaluate the results using InceptionV3_TF and have fallen back on the Torch version which I'm assuming gives results in the same ballpark as the Tensorflow version. With that said, here are my best results on CIFAR10 with ReACGAN-ADC-DiffAug and ReACGAN-DiffAug:

ReACGAN + ADC + DiffAug: (evaluated at best checkpoint 198000/200000)

ReACGAN + DiffAug: (evaluated at best checkpoint 198000/200000)

Both of which seem to conform to my expectations.

Adversarian commented 1 year ago

It might be possible that applying DRA can guide the results to our expectations as per the claims of the authors about how DRA can alleviate the stability and mode collapse issues with a regret minimization objective. I'm not entirely sure since I'm not very well versed into the theory that goes into the DRA regularization algorithm and how it might affect these particular GAN architectures, but I will try to re-do my experiments while applying DRA to both configurations if I have time.

Adversarian commented 1 year ago

I just finished training both models on Tiny ImageNet and the differences are even more stark. Although plain ReACGAN collapsed after about 100k steps, it still achieved far better results than ReACGAN + ADC after 200k steps.

It seems, as you pointed out earlier, there is some correlation between the increase in the number of classes (200 in the case of Tiny ImageNet for instance) and how far these two configurations diverge.