search

georgeyiasemis / Recurrent-Neural-Networks-from-scratch-using-PyTorch

LSTM, RNN and GRU implementations using Pytorch
MIT License
57 stars 12 forks source link

Run time statistics #1

Open Sherrylife opened 5 months ago

Sherrylife commented 5 months ago

Hi, @georgeyiasemis, it is nice to see you doing such a great job. I was wondering if you have compared your implementation and the official implementation of pytorch? I tested it and it seemed that your implementation of LSTM is 10x slower than the official implementation of LSTM using pytorch under the same input. Is there any way to improve the speed in your implementation?

benisalla commented 1 month ago

Could you provide the snippet of code you have tried? It seems to me that there is something incorrect or missing in its implementation.

At least, if you could provide just the hyperparameters, I could compare them and perhaps we could fix the implementation.

Sherrylife commented 1 month ago

Hi, @benisalla , I am sorry that I can't find the snippet of code I have tried since it has been a long time. I remember the code in this repository is similar to other codes in github aiming to implement RNN from scratch and I think you can find the script from others repositories to run them. Besides, I have looked at the codes from these and I remember that some of their implementation do have wrongs, e.g., they can't be used to run 2 or more loop.

Finally, it seems that it is unsuitable to run RNN with Pytorch using your own implementation since it is very slow. If you have read the source code in Pytorch, you will find that there are many optimization tricks to speed up RNN, especially need to use C/C++ and write CUDA codes. If you are interested in it, maybe the following blog can help you.

https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/