Request to support RWKV and Mamba SSMs

unslothai / unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

12.37k stars 805 forks source link

Request to support RWKV and Mamba SSMs #410

Open asphytheghoul opened 2 months ago

asphytheghoul commented 2 months ago

Hello, I have been using unsloth for my fine-tuning purposes and am really enjoying the framework so far! I just wanted to know if you could add support for loading and training state space models like Mamba and the RWKV models as well.

danielhanchen commented 2 months ago

Oh thanks! Interesting I'll take a lot