The MACs result doesn't contain q@k and attn@V

czg1225 / SlimSAM

SlimSAM: 0.1% Data Makes Segment Anything Slim

Apache License 2.0

248 stars 14 forks source link

The MACs result doesn't contain q@k and attn@V #18

Open Yiman-GO opened 1 month ago

Yiman-GO commented 1 month ago

The calculation workload of matrix multiplication in the attention modules(qk and attnV) which is nonnegligible, is not included in torch_pruning.utils.count_ops_and_params。 torchprofile can calculate the macs of q@k and k@v.

czg1225 commented 1 month ago

Hi @Yiman-GO , Thanks for the issue. I will check it with the author of torch pruning.