search

ajbrock / BigGAN-PyTorch

The author's officially unofficial PyTorch BigGAN implementation.
MIT License
2.84k stars 470 forks source link

Singular value clamping code? #57

Closed johntiger1 closed 4 years ago

johntiger1 commented 4 years ago

I am looking for the code that does the clamping of the singular values from the weight matrices: image

(i.e. page 6 of the arxiv paper)

but can't seem to find it in the training loop. Does anyone know where it is?

Thanks

ajbrock commented 4 years ago

This is not included in BigGAN-PyTorch. As that section indicates, BigGAN collapse is not successfully mitigated by any one of a variety of normalization, clipping, or clamping strategies, even though they successfully treat the symptoms of spectral explosion. (Spectral Explosion is also probably an attack from a ghost anime). If you want to try SV clamping for your own experiments, all the tools are provided if you subclass an SN layer and implement eq (4).

johntiger1 commented 4 years ago

Wow, thanks for the quick reply! Sure, but computing the singular values at every single update is the main challenge I would say. Was hoping to find the Arnoldi iteration method in Pytorch somewhere

EDIT: seems like your code in https://github.com/ajbrock/BigGAN-PyTorch/blob/a5557079924c3070b39e67f2eaea3a52c0fb72ab/layers.py

is most of the way there! Thanks!