Accuracy scores / tensorboard logs

[tals]

Hey there. Thanks for publishing this intriguing work. Love the idea!

I'm trying to reproduce your results on BigGAN128, but I think I am missing some details, as the results are weaker than what was noted in the paper.

It would really helpful to have the accuracy scores of the latent shift predictor for each model, or, even better, the tensorboard logs.

Thanks!

[anvoynov]

Hi @tals, thank you for interest! Here are the tensorboard logs: https://drive.google.com/file/d/17VvQTZf3P8MHR6aJ0wo_B7HjqaOOUfhL/view?usp=sharing (I have removed images from logs as they are memory consuming). Also, could you specify, how do you run the code? (Examples are coming soon, my bad)

[tals]

Thanks for the answer! The tensorboard files seem to be missing the accuracy metric, which I was really interested in. Do you happen to have it? Also, is this the BigGAN128 run?

I've done two runs. First one was

run_train.py --out OUT --gan_type BigGAN --batch_size 16 --n_steps 100000

(Note the halved batch size, done due to memory constraints on the Titan V I was using)

I then added gradient accumulation and ran it for 2x as many steps

run_train.py --out OUT --gan_type BigGAN --batch_size 16 --agg_batch_size 32 --n_steps 200000

[anvoynov]

"percent" is actually the shifted direction classifier accuracy.

(Note the halved batch size, done due to memory constraints on the Titan V I was using) I then added gradient accumulation and ran it for 2x as many steps

Have you decreased the learning rate x0.5? We have seen quality degradation on the PGGAN with small batch size (6, or smth around), maybe this is the case here as well. I will try to run the code with your options as well.

[tals]

I didn't for the first one, but the second one did accumulation so it is mostly the same (minus batchnorms stats)

The accuracy seem similar (0.853 here vs 0.85 on yours):

Looking more closely at the discovered latents, I did see a lot of similar things to the paper:

"thickness"

But there is clear entanglement with the background.

"rotation"

It does seem to fall off the manifold at the edges of the shift. I expect one of the deformation loss in the code to help there, but the paper doesn't really talk about it, and the tensorboard files you posted didn't use it either. Have you seen them improve things in your experiments? Another option is to add the pretrained discriminator as an additional loss, to help keep these latents on the manifold (won’t work on DeepMind BigGAN models since they didn’t release the discriminator)

Interestingly, I'm also seeing a lot of similar directions, which seems strange since it would make it hard to guess the chosen direction, yet it doesnt seem to harm accuracy much:

[anvoynov]

I expect one of the deformation loss in the code to help there, but the paper doesn't really talk about it, and the tensorboard files you posted didn't use it either.

could you specify, which term do you mean?

[tals]

could you specify, which term do you mean?

One of the the DeformatorLoss. The code has L2/Relative/Stat (maybe KL against a unit gaussian would work?) but the paper makes no mention of it. https://github.com/anvoynov/GANLatentDiscovery/blob/8fbb3da0ba5bb30d1106922acacdd2eeac0c2620/trainer.py#L214

[anvoynov]

The original purpose of these terms was to regularize deformators of types DeformatorType.FC and DeformatorType.LINEAR https://github.com/anvoynov/GANLatentDiscovery/blob/8fbb3da0ba5bb30d1106922acacdd2eeac0c2620/latent_deformator.py#L10 to avoid them to have large singular values. That's why I'm not quite sure it can help here. We also considered a term that encourages the shifted images to be similar to the original (namely, in terms of lpips), though there were no notable improvements with it. Though playing with it can help in yours "thickness" direction post-fitting.