search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.16k stars 3.99k forks source link

[Bug]: Support Falcon Mamba #7478

Open hahmad2008 opened 1 month ago

hahmad2008 commented 1 month ago

Your current environment

Does VLLM support Falcon Mamba models? if not, when it will be supported

🐛 Describe the bug

Does VLLM support Falcon Mamba models? if not, when it will be supported

robertgshaw2-neuralmagic commented 1 month ago

cc @tlrmchlsmth

Pinkypoopoo commented 1 month ago

Unsubscribe

On Wed, 14 Aug, 2024, 1:37 am Robert Shaw, @.***> wrote:

cc @tlrmchlsmth https://github.com/tlrmchlsmth

— Reply to this email directly, view it on GitHub https://github.com/vllm-project/vllm/issues/7478#issuecomment-2287037438, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCU643RS6ZZAY5MXB4JRVXLZRJRR3AVCNFSM6AAAAABMO6JHLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBXGAZTONBTHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

tlrmchlsmth commented 1 month ago

Hey @hahmad2008,

No, vLLM doesn't support Falcon Mamba yet. I have a work-in-progress PR (https://github.com/vllm-project/vllm/pull/6484) to add support for Mamba. I'll look into supporting FalconMamba as well.

hahmad2008 commented 1 month ago

Thank @tlrmchlsmth. Do you have any idea when this PR will be merged?

hahmad2008 commented 3 weeks ago

@tlrmchlsmth Do you have any idea when this PR will be merged?

tlrmchlsmth commented 3 weeks ago

Hi @hahmad2008, I’ve been prioritizing that PR over the last couple of days and I think will land it later this week. Do note that I’m not planning to add FalconMamba to #6484, but if it’s similar enough to Mamba or Mamba2, there will be a fast follow PR for support

hahmad2008 commented 3 weeks ago

Thank you @tlrmchlsmth. I can only load and apply inference from it using transformers version 4.45.0.dev0 which is still not released. I installed it using: 

pip install -U git+https://github.com/huggingface/transformers.git

So do you think your PR handle this?

tlrmchlsmth commented 3 weeks ago

@hahmad2008 sorry, haven't gotten a chance to look at FalconMamba yet -- If transformers 4.45 is needed, then I'll likely I'll wait for that release.

hahmad2008 commented 2 weeks ago

@tlrmchlsmth Thanks! seems it will be released next week. https://github.com/huggingface/transformers/issues/33236#issuecomment-2324529754