InternVL−Chat−V1.5-Int8 inference is slow

[tairen99]

Hi @all Thank you for your good work!

I notice using the InternVL−Chat−V1.5-Int8 model, the inference time is very slow, as mentioned in link

Do you support using lmdeploy to improve the inference speed or or any method to improve the inference speed?

Thanks in advance.

[BIGBALLON]

Using the 4-bit model quantized by AWQ is recommended, which is very fast and occupies less GPU memory than int8.

• https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5-AWQ

from lmdeploy import pipeline
from lmdeploy.messages import TurbomindEngineConfig
from lmdeploy.vl import load_image

model = 'OpenGVLab/InternVL-Chat-V1-5-AWQ'
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
backend_config = TurbomindEngineConfig(model_format='awq')
pipe = pipeline(model, backend_config=backend_config, log_level='INFO')
response = pipe(('describe this image', image))
print(response)

or service

lmdeploy serve api_server OpenGVLab/InternVL-Chat-V1-5-AWQ --backend turbomind --model-format awq