As far as I know, frozen layers have no effect on FLOPs in the forward pass. That is, even if frozen layers are included, FLOPs is only affected by the total number of parameters.
So, how do estimate the FLOPs in the backward pass when there are some frozen layers im the model? Is it correct to simply calculate 2 * forward_FLOPs?
I wonder if this code reflects my question.
If not, can someone please help me?
juliannadvl
commented
1 year ago
Thank you so much for sharing the code.
As far as I know, frozen layers have no effect on FLOPs in the forward pass. That is, even if frozen layers are included, FLOPs is only affected by the total number of parameters.
So, how do estimate the FLOPs in the backward pass when there are some frozen layers im the model? Is it correct to simply calculate
2 * forward_FLOPs?I wonder if this code reflects my question. If not, can someone please help me?