search

OpenGVLab / VideoMamba

VideoMamba: State Space Model for Efficient Video Understanding
https://arxiv.org/abs/2403.06977
Apache License 2.0
660 stars 47 forks source link

Questions about computing FLOPs with fvcore #18

Closed exhyy closed 3 months ago

exhyy commented 3 months ago

Hi all, thanks for your great work! I am wandering how to compute the FLOPs reported in https://github.com/OpenGVLab/VideoMamba/blob/main/videomamba/image_sm/MODEL_ZOO.md. Take videomamba_tiny as example, I have tried running python videomamba/image_sm/models/videomamba.py with changing the model to videomamba_tiny. Then I got the computed FLOPs, 1.08G, which equals to the reported one. However, if I use the following piece of code:

model = videomamba_tiny().cuda()
flops = FlopCountAnalysis(model, torch.rand(1, 3, img_size, img_size).cuda())
print(flops.by_operator())

I will get some warnings:

Unsupported operator aten::add encountered 72 time(s)
Unsupported operator aten::mul encountered 95 time(s)
Unsupported operator aten::mul_ encountered 24 time(s)
Unsupported operator aten::exp encountered 48 time(s)
Unsupported operator aten::neg encountered 48 time(s)
Unsupported operator prim::PythonOp.MambaInnerFnNoOutProj encountered 48 time(s)
Unsupported operator aten::flip encountered 48 time(s)
Unsupported operator aten::rand encountered 23 time(s)
Unsupported operator aten::floor_ encountered 23 time(s)
Unsupported operator aten::div encountered 23 time(s)
The following submodules of the model were never called during the trace of the graph. They may be unused, or they were accessed by direct calls to .forward() or via other python methods. In the latter case they will have zeros for statistics, though their statistics will still contribute to their parent calling module.
layers.0.mixer.act, layers.0.mixer.conv1d, layers.0.mixer.conv1d_b, layers.0.mixer.dt_proj, layers.0.mixer.dt_proj_b, layers.0.mixer.in_proj, layers.0.mixer.out_proj, layers.0.mixer.x_proj, layers.0.mixer.x_proj_b, layers.1.mixer.act, layers.1.mixer.conv1d, layers.1.mixer.conv1d_b, layers.1.mixer.dt_proj, layers.1.mixer.dt_proj_b, layers.1.mixer.in_proj, layers.1.mixer.out_proj, layers.1.mixer.x_proj, layers.1.mixer.x_proj_b, layers.10.mixer.act, layers.10.mixer.conv1d, layers.10.mixer.conv1d_b, layers.10.mixer.dt_proj, layers.10.mixer.dt_proj_b, layers.10.mixer.in_proj, layers.10.mixer.out_proj, layers.10.mixer.x_proj, layers.10.mixer.x_proj_b, layers.11.mixer.act, layers.11.mixer.conv1d, layers.11.mixer.conv1d_b, layers.11.mixer.dt_proj, layers.11.mixer.dt_proj_b, layers.11.mixer.in_proj, layers.11.mixer.out_proj, layers.11.mixer.x_proj, layers.11.mixer.x_proj_b, layers.12.mixer.act, layers.12.mixer.conv1d, layers.12.mixer.conv1d_b, layers.12.mixer.dt_proj, layers.12.mixer.dt_proj_b, layers.12.mixer.in_proj, layers.12.mixer.out_proj, layers.12.mixer.x_proj, layers.12.mixer.x_proj_b, layers.13.mixer.act, layers.13.mixer.conv1d, layers.13.mixer.conv1d_b, layers.13.mixer.dt_proj, layers.13.mixer.dt_proj_b, layers.13.mixer.in_proj, layers.13.mixer.out_proj, layers.13.mixer.x_proj, layers.13.mixer.x_proj_b, layers.14.mixer.act, layers.14.mixer.conv1d, layers.14.mixer.conv1d_b, layers.14.mixer.dt_proj, layers.14.mixer.dt_proj_b, layers.14.mixer.in_proj, layers.14.mixer.out_proj, layers.14.mixer.x_proj, layers.14.mixer.x_proj_b, layers.15.mixer.act, layers.15.mixer.conv1d, layers.15.mixer.conv1d_b, layers.15.mixer.dt_proj, layers.15.mixer.dt_proj_b, layers.15.mixer.in_proj, layers.15.mixer.out_proj, layers.15.mixer.x_proj, layers.15.mixer.x_proj_b, layers.16.mixer.act, layers.16.mixer.conv1d, layers.16.mixer.conv1d_b, layers.16.mixer.dt_proj, layers.16.mixer.dt_proj_b, layers.16.mixer.in_proj, layers.16.mixer.out_proj, layers.16.mixer.x_proj, layers.16.mixer.x_proj_b, layers.17.mixer.act, layers.17.mixer.conv1d, layers.17.mixer.conv1d_b, layers.17.mixer.dt_proj, layers.17.mixer.dt_proj_b, layers.17.mixer.in_proj, layers.17.mixer.out_proj, layers.17.mixer.x_proj, layers.17.mixer.x_proj_b, layers.18.mixer.act, layers.18.mixer.conv1d, layers.18.mixer.conv1d_b, layers.18.mixer.dt_proj, layers.18.mixer.dt_proj_b, layers.18.mixer.in_proj, layers.18.mixer.out_proj, layers.18.mixer.x_proj, layers.18.mixer.x_proj_b, layers.19.mixer.act, layers.19.mixer.conv1d, layers.19.mixer.conv1d_b, layers.19.mixer.dt_proj, layers.19.mixer.dt_proj_b, layers.19.mixer.in_proj, layers.19.mixer.out_proj, layers.19.mixer.x_proj, layers.19.mixer.x_proj_b, layers.2.mixer.act, layers.2.mixer.conv1d, layers.2.mixer.conv1d_b, layers.2.mixer.dt_proj, layers.2.mixer.dt_proj_b, layers.2.mixer.in_proj, layers.2.mixer.out_proj, layers.2.mixer.x_proj, layers.2.mixer.x_proj_b, layers.20.mixer.act, layers.20.mixer.conv1d, layers.20.mixer.conv1d_b, layers.20.mixer.dt_proj, layers.20.mixer.dt_proj_b, layers.20.mixer.in_proj, layers.20.mixer.out_proj, layers.20.mixer.x_proj, layers.20.mixer.x_proj_b, layers.21.mixer.act, layers.21.mixer.conv1d, layers.21.mixer.conv1d_b, layers.21.mixer.dt_proj, layers.21.mixer.dt_proj_b, layers.21.mixer.in_proj, layers.21.mixer.out_proj, layers.21.mixer.x_proj, layers.21.mixer.x_proj_b, layers.22.mixer.act, layers.22.mixer.conv1d, layers.22.mixer.conv1d_b, layers.22.mixer.dt_proj, layers.22.mixer.dt_proj_b, layers.22.mixer.in_proj, layers.22.mixer.out_proj, layers.22.mixer.x_proj, layers.22.mixer.x_proj_b, layers.23.mixer.act, layers.23.mixer.conv1d, layers.23.mixer.conv1d_b, layers.23.mixer.dt_proj, layers.23.mixer.dt_proj_b, layers.23.mixer.in_proj, layers.23.mixer.out_proj, layers.23.mixer.x_proj, layers.23.mixer.x_proj_b, layers.3.mixer.act, layers.3.mixer.conv1d, layers.3.mixer.conv1d_b, layers.3.mixer.dt_proj, layers.3.mixer.dt_proj_b, layers.3.mixer.in_proj, layers.3.mixer.out_proj, layers.3.mixer.x_proj, layers.3.mixer.x_proj_b, layers.4.mixer.act, layers.4.mixer.conv1d, layers.4.mixer.conv1d_b, layers.4.mixer.dt_proj, layers.4.mixer.dt_proj_b, layers.4.mixer.in_proj, layers.4.mixer.out_proj, layers.4.mixer.x_proj, layers.4.mixer.x_proj_b, layers.5.mixer.act, layers.5.mixer.conv1d, layers.5.mixer.conv1d_b, layers.5.mixer.dt_proj, layers.5.mixer.dt_proj_b, layers.5.mixer.in_proj, layers.5.mixer.out_proj, layers.5.mixer.x_proj, layers.5.mixer.x_proj_b, layers.6.mixer.act, layers.6.mixer.conv1d, layers.6.mixer.conv1d_b, layers.6.mixer.dt_proj, layers.6.mixer.dt_proj_b, layers.6.mixer.in_proj, layers.6.mixer.out_proj, layers.6.mixer.x_proj, layers.6.mixer.x_proj_b, layers.7.mixer.act, layers.7.mixer.conv1d, layers.7.mixer.conv1d_b, layers.7.mixer.dt_proj, layers.7.mixer.dt_proj_b, layers.7.mixer.in_proj, layers.7.mixer.out_proj, layers.7.mixer.x_proj, layers.7.mixer.x_proj_b, layers.8.mixer.act, layers.8.mixer.conv1d, layers.8.mixer.conv1d_b, layers.8.mixer.dt_proj, layers.8.mixer.dt_proj_b, layers.8.mixer.in_proj, layers.8.mixer.out_proj, layers.8.mixer.x_proj, layers.8.mixer.x_proj_b, layers.9.mixer.act, layers.9.mixer.conv1d, layers.9.mixer.conv1d_b, layers.9.mixer.dt_proj, layers.9.mixer.dt_proj_b, layers.9.mixer.in_proj, layers.9.mixer.out_proj, layers.9.mixer.x_proj, layers.9.mixer.x_proj_b
1079579328
Counter({'matmul': 697171968, 'linear': 348777984, 'conv': 28901376, 'layer_norm': 4728000})

I don't understand how fvcore computes FLOPs, but these warnings and the result of flops.by_operator() seem to indicate that this calculation method is not correct.

Andy1621 commented 3 months ago

Good question! Since I'm also unfamiliar with how fvcore runs, maybe you can check how VMamba calculates GFLOPs manually.

exhyy commented 3 months ago

Good question! Since I'm also unfamiliar with how fvcore runs, maybe you can check how VMamba calculates GFLOPs manually.

Thanks for your replay. I just found the answer at https://github.com/state-spaces/mamba/issues/110, which is used by VMamba.