Open DavideHe opened 1 year ago
when I get the macs with thop ,I find the op's macs of torch.matmul will is 0, but it is very heavy macs in self attention of transformer
torch.matmul
self attention of transformer
class M(nn.Module): def __init__(self): super().__init__() def forward(self,x): out = torch.matmul(x,x.transpose(-1, -2)) print(out.shape) return out
when I get the macs with thop ,I find the op's macs of
torch.matmul
will is 0, but it is very heavy macs inself attention of transformer