Stabilizer state missing from sLSTM

muditbhargava66 / PyxLSTM

Efficient Python library for Extended LSTM with exponential gating, memory mixing, and matrix memory for superior sequence modeling.

https://pyxlstm.readthedocs.io/

MIT License

190 stars 17 forks source link

Open Strawl opened 1 month ago

Strawl commented 1 month ago

The new sLSTM doesn't have the stabilizer state m. This leads to exploding gradients very easily.

yanpeng0520 commented 3 weeks ago

Also the normalizer is not implemented?