Closed phatjkk closed 7 months ago
Is there any problem of employing: https://huggingface.co/docs/transformers/quantization#quantization and/or: https://github.com/casper-hansen/AutoAWQ with limited computing resources?
Yeah I have used AutoAWQ version 0.1.7 (which support Bloom) for quantization PhoGPT-7B5-Instruct to 4 bit GEMM group-size 128 and found this error:
The AutoAWQ working well with vietcuna-7b-v3 (which used Bloom too).
Here is the code I have used: https://github.com/phatjkk/test_llm/blob/main/FINALL_GENERATOR_Quantization.ipynb
I don't know what the problem is. I tried some previous version of transformer but may not work! Are you try it yet? Thank you!
Not sure why Bloom is mentioned here. This issue was fixed for MPT-type models (including PhoGPT): https://github.com/casper-hansen/AutoAWQ/issues/155 I will spend time looking into it.
I hope that PhoGPT will have an AWQ or GPTQ version for running on low VRAM GPUs. Do you have any plans for quantization? It will make this LLM more popular for students and individual researchers who have limited computing resources. Thank you for your hard work!