Is there any reason you don't use torch.nn.utils.spectral_norm()?

[Zhangyanbo]

Pytorch has it built-in spectral normalization, i.e., the torch.nn.utils.spectral_norm(). Is there any reason you don't use that?

[jarrelscy]

The inbuilt Pytorch spectral norm uses Miyato's version of Spectral Normalisation (for GANs) (https://arxiv.org/pdf/1802.05957.pdf). The iresnet paper specifies that they use Gouk's version of spectral norm ( https://arxiv.org/pdf/1804.04368.pdf) which is implemented slightly differently.

There are several differences but the largest and probably most relevant to us is that for CNNs Miyato regularises the flattened kernel of the CNN instead of computing the "true" spectral norm of the entire function (which naively would require calculating a huge, mostly sparse matrix). Gouk's method provides an efficient way of estimating the spectral norm through the power method (https://en.wikipedia.org/wiki/Power_iteration).

On Fri, Apr 9, 2021 at 4:20 PM Zhang Yanbo @.***> wrote:

Pytorch has it built-in spectral normalization, i.e., the torch.nn.utils.spectral_norm(). Is there any reason you don't use that?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jarrelscy/iResnet/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOEUB7G5G3HQ7DOOHT5Y7LTH2MDDANCNFSM42UKMV4A .

[Zhangyanbo]

Thank you! This is really helpful!