Counting bias term

[noamgot]

Hello I stumbled upon this package as I'm looking for a way to count FLOPs/MACs in my model. I compared the results that I got to another popular package (thop) and noticed that fvcore does not count the bias term as part of the FLOPs count. I wrote a small example to demonstrate this using a convolutional layer (I'm running on Linux CPU, with the latest version of fvcore and pytorch v1.9.1); notice the difference when we set bias=True and how the results are identical when bias=False:

import torch
import torch.nn as nn
from thop import profile
from fvcore.nn import FlopCountAnalysis

class MyModel(nn.Module):

    def __init__(self, bias):
        super().__init__()
        self.conv = nn.Conv2d(3, 10, 3, bias=bias)

    def forward(self, x):
        return self.conv(x)

input_ = torch.randn((1, 3, 32, 32))
for bias in [True, False]:
    print(f"********************** bias = {bias} ***********************")
    model = MyModel(bias=bias)
    print("fvcore result:")
    flops = FlopCountAnalysis(model, input_)
    print(flops.total())
    print("thop results:")
    macs, _ = profile(model, inputs=(input_,))
    print(macs)
    print("***********************************************************")

It outputs:

********************** bias = True ***********************
fvcore result:
[W NNPACK.cpp:79] Could not initialize NNPACK! Reason: Unsupported hardware.
243000
thop results:
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.Conv2d'>.
[WARN] Cannot find rule for <class '__main__.MyModel'>. Treat it as zero Macs and zero Params.
252000.0
***********************************************************
********************** bias = False ***********************
fvcore result:
243000
thop results:
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.Conv2d'>.
[WARN] Cannot find rule for <class '__main__.MyModel'>. Treat it as zero Macs and zero Params.
243000.0
***********************************************************

Process finished with exit code 0

My questions:

1. Is that a bug, or a feature?
2. Is there a way to include the bias in the calculation without modifying the internal code?

Thanks

[ppwwyyxx]

1. It's expected, although arguable. To fvcore's defense, it's counting MACs, so if there is no multiply it counts 0.
2. You can call set_op_handle https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.nn.ActivationCountAnalysis.set_op_handle to overwrite the behavior. And check the handles in source code to see how to implement a handle.

[noamgot]

I'll have a look, thanks!