search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
131.24k stars 26.09k forks source link

Add Model Support for xLSTM #27011

Open stefan-it opened 10 months ago

stefan-it commented 10 months ago

Model description

Inspired by recent rumors about xLSTM - a hidden successor to LSTM - by Sepp Hochreiter, this issue tracks the open source implementation about adding xLSTM to Transformers library.

Open source status

Provide useful links for the implementation

At the moment no implementation does exist.

Only rumors that xLSTM surpasses GPT-2 on various (small) downstream datasets.

Good overview is the xLSTM Resources repository from @AI-Guru.

Pythoniasm commented 10 months ago

Sounds like a money grab. If it is something useful, he should have chosen the academic path or at least filing patent.

This way of boldly claiming success via non-serious media channels is highly unprofessional. It smells like publicity is more relevant than results which further supports motivations like funding/personal gains/politics.

DavidFarago commented 6 months ago

If I understood it correctly, a patent is on its way, and at least a paper about xLSTM will be published in less than 6 month.

KnutJaegersberg commented 6 months ago

I have some doubts if this is planned as an open source model.

albertz commented 3 months ago

Paper is published now: https://arxiv.org/abs/2405.04517

Ghost---Shadow commented 2 months ago

Need code and checkpoint or it didn't happen.

stefan-it commented 2 months ago

Official implementation is out now:

https://github.com/NX-AI/xlstm

danthe1st commented 2 months ago

Note that the official source code is AGPL-licensed.