Closed pykeras closed 2 weeks ago
Hi, due to significant quantization errors with BNB 4-bit quantization on InternViT-6B, the model may produce nonsensical outputs and fail to understand images. Therefore, please avoid using BNB 4-bit quantization.
For these AWQ models we publish, you need to use them with lmdeploy, see here.
If you do not want to use lmdeploy, you can wait until we update the vllm version later.
Hi, due to significant quantization errors with BNB 4-bit quantization on InternViT-6B, the model may produce nonsensical outputs and fail to understand images. Therefore, please avoid using BNB 4-bit quantization.
For these AWQ models we publish, you need to use them with lmdeploy, see here.
If you do not want to use lmdeploy, you can wait until we update the vllm version later.
thanks for reply, the main issue with lmdeploy
is the amount of GPU it use.
Checklist
Describe the bug
I'm using the InternVL2.0 model in 4bit mode with low_cpu_mem_usage enabled, following the instructions in the Hugging Face ReadMe. However, regardless of the input prompt or image, the model consistently returns nonsensical text.
Issue Summary: The InternVL2.0 model is not producing expected output when used in AWQ format. The model should be able to generate coherent and relevant responses to input prompts or images, but instead it is returning random characters and phrases.
Reproduction
Steps to Reproduce: Clone the Hugging Face repository for InternVL2.0.
OpenGVLab/InternVL2-26B-AWQ
orOpenGVLab/InternVL2-8B-AWQ
Install the required dependencies using pip install :Load the model in 4bit mode with low_cpu_mem_usage enabled.
Pass a prompt with image to the model and observe the output.
Example output
Environment
Error traceback
No response