search

bd-iaas-us / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
3 stars 1 forks source link

[feature] GGUF support #8

Closed thesues closed 1 month ago

thesues commented 5 months ago

🚀 The feature, motivation and pitch

GUFF should be supported as well.

Alternatives

No response

Additional context

No response

thesues commented 4 months ago

code to consider: https://github.com/PygmalionAI/aphrodite-engine/pull/320/files

chizhang118 commented 2 months ago

This is GGUF, not GUFF.

thesues commented 1 month ago

upstream accepted another one