[Feature request] Support loading GGUF and GGML model format

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

Apache License 2.0

36.47k stars 4.49k forks source link

Open nghidinhit opened 1 year ago

merrymercy commented 1 year ago

Community contributions are welcome

digisomni commented 5 months ago

Does this work via SGLang? If not, I think we should just push support for it through there instead of the native model runner here?

surak commented 3 months ago

@digisomni If you would add new functionality, it would be nice to have first on the default worker.

02shanks commented 1 month ago

@surak @merrymercy is this still open for contribution?

surak commented 1 week ago

Absolutely!