Control over latent interpolations

[adrienchaton]

Hi all,

I am training style GAN 3 on my own data, some organic shots. I used to train style GAN 2 ADA too.

I am trying to understand a bit the added parameters to the generation of style GAN 3, still some things I am puzzled about. This applies both to a model trained with R equivariance or T equivariance, according to the default configs.

When I random sample a fixed z and move across a certain range of translate and rotate to provide to make_transform (gen_images.py) then it seems I get more or less the same image and indeed it is just translated or rotated. So equivariance seems working fine.

Now I keep translate and rotate fixed, but instead do a linear interpolation in between two random z. With SGAN2 I would get a smooth morphing and the point of view seems fixed, i.e. everything stays in the center crop of the dataset. With SGAN3 it seems there is a mix between morphing and also some movements of the point of view.

Is this "expected" or would there be a way to have z control the morphing only and make_transform control the view point only ?

I attach two videos of linear interpolation, same training data but either T or R config for the model.

https://user-images.githubusercontent.com/35500385/154847473-ec78dd12-18b9-4c44-956b-e04ee46d4ee7.mp4

https://user-images.githubusercontent.com/35500385/154847488-2b66c3e0-5e60-49a5-a1db-ae8db233ee7e.mp4

Thanks !

[PDillis]

There is an 'easy' fix: anchor the latent space to w_avg. A video example: https://twitter.com/PDillis/status/1448254764593123330

I put the code in the next tweet with attribution, but I'll repeat it here for ease of access:

# Thanks to @RiversHaveWings and @nshepperd1
if hasattr(G.synthesis, 'input'):
    shift = G.synthesis.input.affine(G.mapping.w_avg.unsqueeze(0))
    G.synthesis.input.affine.bias.data.add_(shift.squeeze(0))
    G.synthesis.input.affine.weight.data.zero_()

You could add this right after loading G, for example.

[adrienchaton]

Thank you for sharing ! I will try that soon :)