What about GPTQ v2 format without groupsize?

johnsmith0031 / alpaca_lora_4bit

MIT License

534 stars 84 forks source link

What about GPTQ v2 format without groupsize? #65

Closed Ph0rk0z closed 1 year ago

Ph0rk0z commented 1 year ago

I have tried to get things to load for GPTQv2 models that do not use groupsize. Appears it is impossible because groupsize is used to switch between V1 and V2.

johnsmith0031 commented 1 year ago

updated cuda implementation to support g_idx. I think it'll work now.

Ph0rk0z commented 1 year ago

Working great now that I updated the cuda kernel too :)

Thank you!