BASED - Githubissues

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

https://huggingface.co/transformers

Apache License 2.0

133.19k stars 26.6k forks source link

BASED #29466

Open axelmagn opened 7 months ago

axelmagn commented 7 months ago

Model description

BASED is an attention model which combines sliding window attention and global linear attention to capture similar dependencies to transformers in a subquadratic model.

It outperforms other similar models such as Mamba.

Open source status

[X] The model implementation is available
[X] The model weights are available

Provide useful links for the implementation

GitHub Repository
Model Weights: 360m, 1b

simran-arora commented 6 months ago

Hi I'm curious if it's possible to add in these models. Is there anything I can do to speed it along?