search

lmnt-com / haste

Haste: a fast, simple, and open RNN library
Apache License 2.0
325 stars 27 forks source link

Support zoneout on lstm cell state and add recurrent dropout #29

Open azraelkuan opened 3 years ago

azraelkuan commented 3 years ago

hi，any plan about these two questions

  1. lstm zoneout on cell state the same with hidden state https://github.com/lmnt-com/haste/blob/9da2454584d5b5bc9b2ae84a3fa2b271306ec622/frameworks/pytorch/layer_norm_lstm.py#L62-L66

  2. add recurrent dropout the same with keras https://github.com/tensorflow/tensorflow/blob/fcc4b966f1265f466e82617020af93670141b009/tensorflow/python/keras/layers/recurrent.py#L2450-L2459

thanks!

sharvil commented 3 years ago

Do you have a specific use-case in mind for these two additions? I'm open to adding these features – it would be nice to have a sense of the scope and value of these changes.

azraelkuan commented 3 years ago

@sharvil oh，thanks to your reply. for the first question: in the original paper, it has two different zonout on lstm states. for the second question: the dropout will be benifit for very long sequence. like tts acoustic modeling