Closed tairen99 closed 1 month ago
Using the 4-bit model quantized by AWQ is recommended, which is very fast and occupies less GPU memory than int8.
from lmdeploy import pipeline
from lmdeploy.messages import TurbomindEngineConfig
from lmdeploy.vl import load_image
model = 'OpenGVLab/InternVL-Chat-V1-5-AWQ'
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
backend_config = TurbomindEngineConfig(model_format='awq')
pipe = pipeline(model, backend_config=backend_config, log_level='INFO')
response = pipe(('describe this image', image))
print(response)
or service
lmdeploy serve api_server OpenGVLab/InternVL-Chat-V1-5-AWQ --backend turbomind --model-format awq
Hi @all Thank you for your good work!
I notice using the InternVL−Chat−V1.5-Int8 model, the inference time is very slow, as mentioned in link
Do you support using lmdeploy to improve the inference speed or or any method to improve the inference speed?
Thanks in advance.