How to calculate flops spent for forward/backward pass?

[mrgloom]

How to calculate flops spent for forward/backward pass?

For example we have two machines one have some old GPU and another one - modern GPU, so when we train some model (say AlexNet) we have different time spent for each iteration, but I want to know how many flops each machine spent.

Can we caclulate this "statically" using just layers definition or it's more complicated?

[shadjis]

Hi mrgloom,

You can approximate the number of operations in AlexNet as the sum of the number of operations in all convolution and FC layer GEMM (matrix multiply) operations. So e.g. in the forward pass of AlexNet, this is 5 matrix multiplies for conv and 3 for FC. Also note that this number of operations is independent of the device you are using, so you can also just calculate the ratio of their throughput as the ratio of the time without counting the total number of operations. The number of operations is also the same as it would be for Caffe, so we just report the time. CcT currently does not print the number of operations, but I will look into this as an enhancement.

[mrgloom]

What I want to do is to set user a "rating"(like in torrents, or in SETI-HOME) depending on how much FLOPS he spent.

I want to know if is it possible to compute total FLOPS that will be spent for forward/backward pass only using *.prototxt before running caffe?

Also it seems there can be different method of calculation convolution (FFT, CuDNN, etc.) so complexity of operation also will be different?

[deryag]

Hi mrgloom, Could you calculate FLOPS "statically" using just layers definition? I also want to make this calculation for Lenet.

[mrgloom]

Seems it possible look at http://dgschwend.github.io/netscope/quickstart.html For example http://dgschwend.github.io/netscope/#/preset/alexnet

[deryag]

Thank you very much mrgloom, its very easy and understandble.