[Feature]: Support More Models in QLoRA of VLLM

bd-iaas-us / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

https://docs.vllm.ai

Apache License 2.0

1 stars 0 forks source link

[Feature]: Support More Models in QLoRA of VLLM #13

Open chenqianfzh opened 3 weeks ago

chenqianfzh commented 3 weeks ago

🚀 The feature, motivation and pitch

Support models like GptNeo

Alternatives

No response

Additional context

No response

skykiseki commented 3 weeks ago

Hi，I am pleased to see that vLLM has started to support Qlora. May I ask if the current version 0.5.0 only supports the llama-1 model or does it support the entire llama series of models?

thesues commented 1 week ago

such as this: https://huggingface.co/unsloth/mistral-7b-bnb-4bit/tree/main should be supported?