autumnai / leaf

Open Machine Intelligence Framework for Hackers. (GPU/CPU)
Apache License 2.0
5.56k stars 270 forks source link

Implement RNN/LSTMs for Leaf #73

Open MichaelHirn opened 8 years ago

MichaelHirn commented 8 years ago

This issue makes part of #20 more concrete.

Recurrent Neural Networks, became an effective Neural Network architecture, that we would like to implement in Leaf as well. The operations could probably be composed of BLAS operations with collenchyma-blas. Differentiation for backpropagation is another topic, that might better be solved with Auto Differentiation.

johnnyman727 commented 8 years ago

@MichaelHirn if this isn't super timer critical, I can own this task. I will need some hand-holding to get started. Where would be the best place to start?

MichaelHirn commented 8 years ago

Awesome!

I think it should be verily straight forward. Two resources about RNN and performance, that look interesting and might be helpful for technical implementation.

The rest is just adding a new common layer to Leaf, using the BLAS operations from collenchyma-nn. You can reach @hobofan anytime on the gitter chat for any guidance.

If you succeed we the implementation, we are super thrilled to publish the Benchmarks of your RNN/LSTM implementation on Deep Learning Benchmarks and compare them with RNN implementations of other Frameworks.

MichaelHirn commented 8 years ago

As an update:

cuDNN v5, which will be released for early developers in a few weeks, will come with primitives for LSTM/RNNs.

carlthome commented 8 years ago

Adding most of the optimizations described in cuDNN 5 here would easily make leaf the fastest basic LSTM implementation out there.

robsmith11 commented 7 years ago

Was any progress made on this? I might try to add LSTM support in the future.