search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
39.48k stars 5.19k forks source link

Add support for loading pre-quantized Open-Assistant 4-bit #395

Closed kft334 closed 1 year ago

kft334 commented 1 year ago

Currently https://github.com/0cc4m/GPTQ-for-LLaMa/tree/gptneox has to be manually installed and a few lines have to be modified in webui to load the pre-quantized model. Would be good to automate this to be able to load it like LLaMa. I know that there will be better models but it's the best one that I've been able to run yet and it would make it more accessible.

Ph0rk0z commented 1 year ago

I wonder if his changes can just be merged to GPTQ itself. That would make things easier. Some dataset py changes were only ones that alter GPTQ stuff. Unless the new commits massively change things, it's easier than maintaining a separate repo.

github-actions[bot] commented 1 year ago

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.