search

Lyken17 / pytorch-OpCounter

Count the MACs / FLOPs of your PyTorch model.
MIT License
4.9k stars 528 forks source link

count_normalization is only correct for batch_norm. wrong flops count for layernorm #220

Open pzpzpzp2 opened 8 months ago

pzpzpzp2 commented 8 months ago

https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/profile.py#L32 The same count_normalization function is used for every norm-esque module but batchnorms store an estimate mean and stdev, while layernorms calculate them at inference time. Shouldn't layernorms account for the cost of evaluating mean and stdev? The difference is pretty significant: The mean is n flops, stdev is 2n more flops? and thats before the rest of the norm module which is another 2n. Is there a reason layernorms should be estimateable as only 2n flops by re-using batchnorm's estimate?

dyhBUPT commented 4 months ago

I think you are right. BTW, I think the MACs of BN (eval, no affine) should be n, not 2n in codes.

https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/vision/basic_hooks.py#L60-L69

https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/vision/calc_func.py#L43-L45