got 0 ops for nn.MultiheadAttention

[wangtiance]

My thop version: 0.1.1

Minimal code to replicate:

import thop
import torch.nn as nn
import torch

f = nn.MultiheadAttention(100, 1, batch_first=True)

x = torch.ones((1, 1000, 100))

result = thop.profile(f, (x,x,x))
print(result)

I got (0.0, 0). Is this module not supported yet?

[wangtiance]

Does it only support modules listed here? https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/profile.py#L21

[CaA23187]

you are right. I guess this module doesn't supported MHSA layer yet.

[quancs]

facing the same problem

[HaoKang-Timmy]

You may use https://github.com/HaoKang-Timmy/torchanalyse or torchprofile for NLP models

[HaoKang-Timmy]

You may use https://github.com/HaoKang-Timmy/torchanalyse or torchprofile for NLP models

These two repo could profile transformers, which I have tried.

[CaA23187]

You may use https://github.com/HaoKang-Timmy/torchanalyse or torchprofile for NLP models

thank you for your reply. I'll try it!

[cooma04]

It seems thop does not account for any parameters created with nn.Parameter. For any created with nn.Parameter, it simply yields 0.