tonylins
commented
3 years ago Hi, with torchprofile, the MACs are computed on a jit traced graph, which means that the convolution/linear/addition/etc. operations in the customized operators are also counted. There are some customized CUDA kernels (e.g., FusedLeakyReLU) in StyleGAN2, which cannot be captured by torchprofile. Therefore, it is recommended to set FORCE_NATIVE=1 when measuring the FLOPs. But the difference is almost negligible.
Feel free to reopen if you have other questions.
Hi, thanks for your impressive work. I have some problems about MACs/flops.
I compute MACs based on https://github.com/Lyken17/pytorch-OpCounter. However, the computational cost is a little different from your results. The styleganv2 has some custom ops: 'PixelNorm', 'EqualConv2d', 'EqualLinear', 'ModulatedConv2d', 'StyledConv', 'ConvLayer', 'ResBlock', 'ConstantInput', 'ToRGB',(which include FusedLeakyReLU, fused_leaky_relu, upfirdn2d, NoiseInjection, Blur)
I am not sure if you consider all of these ops.
In your code fid.py, I only find
if hvd.rank() == 0: try: from torchprofile import profile_macs macs = profile_macs(generator, torch.rand(1, 1, 512).to(device))
However, the torchprofile package does not contain the above operations.
In content GAN compression https://github.com/lychenyoko/content-aware-gan-compression/blob/master/Util/Calculators.py, it seems that it regards them as CONV and LINEAR layers. Other operations (e.g., FusedLeakyReLU, fused_leaky_relu, upfirdn2d, NoiseInjection, Blur) are ignored.
Could you please let me know how to compute the MACs to reproduce the code? Thanks!