How to count FLOPs?

[miladdona]

Hi,

How can I count number of floating point operation in kerasdense layers in this library (t3f)?

In normal way we do Y = AX + b in a dense layer in keras: Y: output X: input A: weights (in dense layers is a 2-D matrix in shape (m, n)) b: bias *num of FLOPs: m n + b**

But in t3f I don't know how can I count FLOPs?

Thanks

[Bihaqo]

Hi,

You can ask TF about flops count of some function as discussed here: https://github.com/tensorflow/tensorflow/issues/32809

But when I tried using it on the TT layer, it returns zero: https://colab.research.google.com/drive/1LsXKfmEuPLwEgjmSyzS95Xb4qmw4vwfo?usp=sharing

I'll figure it out..

[Bihaqo]

I think this should be about right: https://colab.research.google.com/drive/16S_SUbIjhnQBFj_r7sCpbwZNHADIzEwX?usp=sharing

(Note that the FLOP count in this example is not very impressive because the layer is super small)

[miladdona]

Thanks for your answer. Why do you mentioned in this link https://colab.research.google.com/drive/16S_SUbIjhnQBFj_r7sCpbwZNHADIzEwX?usp=sharing Approximate number of flops of a TT-layer.? Why it is not accurate?!

[Bihaqo]

The bottleneck of the TT-layer is tensor contractions (~= matrix multiplications) and they are counted correctly here. However, there are also much cheaper reshapes and axis transposes and I don't know exactly which and how many of them there are because to implement the layer I use tf.einsum operation which automatically converts tensor contraction rule into a sequence of reshapes / transposes / matrix multiplications. So in the function above I added approximate number of FLOPs that is enough to do one reshape, but this might not be entirely accurate.

[Aelphy]

you can count the flops of einsum using np.einsum_path

[miladdona]

Thank you for your answer.

I had a mistake in the post: *num of FLOPs: m n + b this is num of params in a dense layer and num of flops in a dense layer calculate by : 2 m n anyway, in general I found by using this library we decrease the num of params (memory) and increase the num of flops (speed)**. Am I right?

I have access to this link: https://t3f.readthedocs.io/en/latest/ and the papers mentioned in this link. Do you have any other document or tutorial to understand this library (in depth)?

Thanks. Miladdona

[Aelphy]

depends on the settings, sometimes it is possible to reduce FLOPs as well with really low rank

[miladdona]

Why do you use this statement at the end of for-loop: _input_right *= core_left / coreleft Is this a mistake or no?

[miladdona]

Also, input_left and core_right are equal. This is correct or no?