Open ortegaalfredo opened 1 year ago
try this:
import matmul_utils_4bit
matmul_utils_4bit.faster = False
Not working yet.
File "/home/guest/ai/finetune/alpaca_lora_4bit/matmul_utils_4bit.py", line 131, in matmul4bit output = _matmul4bit_v2(x, qweight, scales, zeros, g_idx) File "/home/guest/ai/finetune/alpaca_lora_4bit/matmul_utils_4bit.py", line 73, in _matmul4bit_v2 quant_cuda.vecquant4matmul(x, qweight, y, scales, zeros, g_idx) RuntimeError: expected scalar type Float but found Half
Try converting all scales to float?
Hi! I'm trying to load the model at https://huggingface.co/Neko-Institute-of-Science/LLaMA-13B-4bit-128g
And it only generates garbage. I'm just doing inference of the model, not training yet. This is an example output:
Any guidance about how to load those models?