search

mlabonne / blog

https://mlabonne.github.io/blog/
https://mlabonne.github.io/blog/
31 stars 9 forks source link

What is a Tensor in Deep Learning? | Maxime Labonne #8

Open utterances-bot opened 1 year ago

utterances-bot commented 1 year ago

What is a Tensor in Deep Learning? | Maxime Labonne

A definition in simple terms with examples

https://mlabonne.github.io/blog/what-is-a-tensor/

hideousparadigm commented 1 year ago

Hy, thanks for the intro. I had a play with this and it is a slighty unfair test as the default dtype in tensor.rand() is float32, wheras numpy uses numpy.float64. I copied the tensor data to numpy arrays and repeated; array1 = tensor1.cpu().detach().numpy() array2 = tensor2.cpu().detach().numpy() and the function clocked at exactly half the time. My i5-4670 took 360 ms (instead of 706ms using float64 values), versus my RTX3060 (4.89 ms) == 74x speed for the fair comparison (that is a 10 year old cpu though:)

Also, using tensors on the CPU, instead of GPU, makeas a more fair test; device = torch.device("cpu") And I got 156ms, which makes tensor 32x faster than equivalent numpy. This just suggests to me that numpy arrays carry a real overhead. Surely there is a way to operate on arrays in python much better than this without using tensors??

Anyway, it was fun watching nvtop as my new gpu was maxed out for a few seconds. Thanks!

mlabonne commented 1 year ago

Hi @hideousparadigm, thanks for your insightful comment, I didn't think about checking that.

I think the float32 vs. float64 is particularly relevant. Storing the tensors on a GPU might not be fair, but this is their main feature and it would be a shame not to use it. However, it's good to know that tensors are still faster than numpy arrays.

Thanks a lot, and I hope you'll upgrade your poor CPU soon! :)