We Need Compile Support For Mamba!

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

https://huggingface.co/transformers

Apache License 2.0

128.63k stars 25.52k forks source link

We Need Compile Support For Mamba! #31246

Open zhenglongjiepheonix opened 1 month ago

zhenglongjiepheonix commented 1 month ago

Feature request

This feature adds torch.compile support for mamba archtecture

Motivation

The motivation is that by supporting compile on mamba, we can get faster inference speed and better throughput even if we don't have high performance specified mamba kernels installed!

Your contribution

I will be implementing this feature

amyeroberts commented 1 month ago

cc @ArthurZucker