Open grumpyp opened 5 months ago
Hi grumpyp , do you solve this issue? i have the same problem.
Hi @Xinyumi I believe I wasn't able to solve the issue. Let me know if you've got any news. My "workaround" was using another package!
Hi @grumpyp Thank you for feedback.I face this issue when i run the benchmark_throughput.py The issue will occur and can not find the config file when i choose the any quantization method except the "fp8". Could you kindly tell the another package which can work?😊
Hi @grumpyp I guess the cause is that the hf model"Mistral-7B-v0.1" has no quant_config , and should be quantizatized used by autoawq before we run the vllm to load model with "awq". You can try the hf model"TheBloke/Llama-2-7b-Chat-AWQ" which can work.
Thanks I might try this if I need to change the currently running version again. Thank you.
Your current environment
How would you like to use vllm
I want to run inference of a fineuned Mistral-7B-v0.1.
Leads to
ValueError: Cannot find the config file for awq
even tho I saw people using it on the same model. Any idea?