Closed Ege-P closed 1 year ago
Triton not found is normal for Windows. (Triton is not compatible with it at the moment, only Linux or WSL2) But based on that error, you have installed the Triton branch of GPTQ. You need to install either Ooba's fork of GPTQ or the CUDA branch of GPTQ from Qwop.
NOTE: Qwop's GPTQ defaults to Triton. I would recommend just using Ooba's for for the moment. https://github.com/oobabooga/text-generation-webui/blob/main/docs/GPTQ-models-(4-bit-mode).md
Yes, this worked perfectly for all the problems from other models as well. I don't know why one-click installer didn't download the ooba branch. (might as well my mistake at some point). Anyway, thank you.
This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.
Describe the bug
I searched in issues and in GitHub pages but couldn't solve this issue. Trying to load a Llama GPTQ model. In parameters, selected: 4 Wbits, 128 groupsize and Llama model type
When I try to load the model, I get this error in the WebUI (also msg in cmd that says: triton not installed).
With the one-click installer, it downloads gptq-for-llama in the repository automatically. If I delete that file, I get another error: No module named ‘llama_inference_offload’
I tried to reinstall from its GitHub page instead of installer.bat but I got the same result.
Is there an existing issue for this?
Reproduction
Download model: https://huggingface.co/Aitrepreneur/wizardLM-7B-GPTQ-4bit-128g/tree/main Try loading in webUI.
Screenshot
No response
Logs
System Info