How to calculate the FLOPs when the model has some freezed variables, or using LoRA?

MrYxJ / calculate-flops.pytorch

The calflops is designed to calculate FLOPs、MACs and Parameters in all various neural networks, such as Linear、 CNN、 RNN、 GCN、Transformer(Bert、LlaMA etc Large Language Model)

https://pypi.org/project/calflops/

MIT License

508 stars 16 forks source link

How to calculate the FLOPs when the model has some freezed variables, or using LoRA? #30

Open Ledzy opened 3 months ago

Ledzy commented 3 months ago

I suppose the corresponding backward calculations should be discarded.