Open kklots opened 1 month ago
使用0.4.2版本,同样的步骤和代码,InternVL-Chat-V1-5-AWQ服务工作正常。
启动server的时候加上 --log-level INFO
,然后把日志发一下吧,包括启动server 和 收到请求。
0.5.0 服务器启动: 都是类似下面的警告 ... [TM][WARNING] Device 1 peer access Device 2 is not available. [TM][WARNING] Device 1 peer access Device 3 is not available. [TM][WARNING] Device 2 peer access Device 0 is not available. [TM][WARNING] Device 2 peer access Device 1 is not available. [TM][WARNING] Device 2 peer access Device 3 is not available. [TM][WARNING] Device 3 peer access Device 0 is not available. [TM][WARNING] Device 3 peer access Device 1 is not available. [TM][WARNING] Device 3 peer access Device 2 is not available. [TM][WARNING] Device 0 peer access Device 1 is not available. [TM][WARNING] Device 0 peer access Device 2 is not available. [TM][WARNING] Device 0 peer access Device 3 is not available. [TM][WARNING] Device 1 peer access Device 0 is not available. [TM][WARNING] Device 1 peer access Device 2 is not available. [TM][WARNING] Device 1 peer access Device 3 is not available. [TM][WARNING] Device 2 peer access Device 0 is not available. [TM][WARNING] Device 2 peer access Device 1 is not available. [TM][WARNING] Device 2 peer access Device 3 is not available. [TM][WARNING] Device 3 peer access Device 0 is not available. [TM][WARNING] Device 3 peer access Device 1 is not available. [TM][WARNING] Device 3 peer access Device 2 is not available. [TM][WARNING] Device 0 peer access Device 1 is not available. [TM][WARNING] Device 0 peer access Device 2 is not available. [TM][WARNING] Device 0 peer access Device 3 is not available. [TM][WARNING] Device 1 peer access Device 0 is not available. [TM][WARNING] Device 1 peer access Device 2 is not available. [TM][WARNING] Device 1 peer access Device 3 is not available. [TM][WARNING] Device 2 peer access Device 0 is not available. [TM][WARNING] Device 2 peer access Device 1 is not available. [TM][WARNING] Device 2 peer access Device 3 is not available. [TM][WARNING] Device 3 peer access Device 0 is not available. [TM][WARNING] Device 3 peer access Device 1 is not available. [TM][WARNING] Device 3 peer access Device 2 is not available. HINT: Please open http://0.0.0.0:8086 in a browser for detailed api usage!!! HINT: Please open http://0.0.0.0:8086 in a browser for detailed api usage!!! HINT: Please open http://0.0.0.0:8086 in a browser for detailed api usage!!! INFO: Started server process [10633] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8086 (Press CTRL+C to quit)
服务端收到请求: INFO: 192.168.102.18:59546 - "GET /v1/models HTTP/1.1" 200 OK INFO: 192.168.102.18:59560 - "POST /v1/chat/completions HTTP/1.1" 200 OK 2024-07-11 04:00:08,818 - lmdeploy - INFO - start ImageEncoder._forward_loop 2024-07-11 04:00:08,818 - lmdeploy - INFO - ImageEncoder received 1 images, left 1 images. 2024-07-11 04:00:08,818 - lmdeploy - INFO - ImageEncoder process 1 images, left 0 images.
0.4.2: 服务端启动,都是类似下方的警告 ... [TM][INFO] Set logger level by INFO [TM][WARNING] Device 1 peer access Device 0 is not available. [TM][WARNING] Device 1 peer access Device 2 is not available. [TM][WARNING] Device 1 peer access Device 3 is not available. [TM][INFO] Set logger level by INFO [TM][WARNING] Device 2 peer access Device 0 is not available. [TM][WARNING] Device 2 peer access Device 1 is not available. [TM][WARNING] Device 2 peer access Device 3 is not available. [TM][INFO] Set logger level by INFO [TM][WARNING] Device 3 peer access Device 0 is not available. [TM][WARNING] Device 3 peer access Device 1 is not available. [TM][WARNING] Device 3 peer access Device 2 is not available. HINT: Please open http://0.0.0.0:8086 in a browser for detailed api usage!!! HINT: Please open http://0.0.0.0:8086 in a browser for detailed api usage!!! HINT: Please open http://0.0.0.0:8086 in a browser for detailed api usage!!! INFO: Started server process [11513] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8086 (Press CTRL+C to quit)
服务端收到请求:
INFO: 192.168.102.18:51190 - "GET /v1/models HTTP/1.1" 200 OK
INFO: 192.168.102.18:51190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2024-07-11 04:04:07,163 - lmdeploy - INFO - ImageEncoder received 1 images, left 1 images.
2024-07-11 04:04:07,163 - lmdeploy - INFO - ImageEncoder process 1 images, left 0 images.
/data2/lixuan/miniconda3/envs/lmdeploy/lib/python3.9/site-packages/torch/utils/checkpoint.py:460: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
/data2/lixuan/miniconda3/envs/lmdeploy/lib/python3.9/site-packages/torch/utils/checkpoint.py:90: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
2024-07-11 04:04:08,692 - lmdeploy - INFO - ImageEncoder forward 1 images, cost 1.529s
2024-07-11 04:04:08,692 - lmdeploy - INFO - ImageEncoder done 1 images, left 0 images.
2024-07-11 04:04:08,699 - lmdeploy - INFO - prompt='<|im_start|>system\nYou are an AI assistant whose name is InternLM (书生·浦语).<|im_end|>\n<|im_start|>user\n
internvl2 也遇到了同样的问题
好像是因为0.5.0版本有些默认参数和0.4.2不一致,你试试启动命令加这几个参数看看
--model-name internvl-internlm2 --session-len 32776 --num-tokens-per-iter 8192
也支持internvl2 我试过8B和26B-AWQ都能用
这个问题好像跟 https://github.com/InternLM/lmdeploy/issues/1981 类似,具体原因我还不清楚,猜测应该跟image encoder的实现跟多线程换成协程有关 lmdeploy/vl/engine.py,出现这个问题的,可以尝试把这一行改成
features = await self.vl_encoder.infer(images) https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/serve/vl_async_engine.py#L66
--session-len 32776 --num-tokens-per-iter 8192 加上这两个参数就好使了
我看了0.5.0版本的启动日志,session_len这个参数默认设置成9w多,我显存最多能到6w,所以就会炸。
仔细对比0.4.2和0.5.0在启动时打印的日志信息就能发现了。还有没有其他的参数变动就不知道了
好像是因为0.5.0版本有些默认参数和0.4.2不一致,你试试启动命令加这几个参数看看
--model-name internvl-internlm2 --session-len 32776 --num-tokens-per-iter 8192
也支持internvl2 我试过8B和26B-AWQ都能用
请问internvl 26B-AWQ在哪里可以下?谢谢
我看了0.5.0版本的启动日志,session_len这个参数默认设置成9w多,我显存最多能到6w,所以就会炸。
仔细对比0.4.2和0.5.0在启动时打印的日志信息就能发现了。还有没有其他的参数变动就不知道了
这个问题在 #2007 解决了,v0.5.1 release,下周
Hi @kklots May you try this https://github.com/zhyncs/lmdeploy-build/releases/tag/aa07f92
Hi @lvhan028 @zhyncs @irexyc Is it possible to use the same deployment method mentioned in this issue to infer a batch of prompts? Just like how we can input a list of [(prompt, image), ...]
into the pipeline
to further speed up the inference.
Hi @lvhan028 @zhyncs @irexyc Is it possible to use the same deployment method mentioned in this issue to infer a batch of prompts? Just like how we can input a list of
[(prompt, image), ...]
into thepipeline
to further speed up the inference.
Yes, you can. Please refer to the guidance mentioned here
The messages
can be a list of request
@lvhan028 Thanks for your response! I followed the guidance and multiplied the messages list by 10 usingmessages = messages * 10
. I expected to get 10 responses but only received one. Did I do something wrong?
Sorry, my bad. I found in vlm scenarios, lmdeploy didn't support the batch message input. Let me check with the team if it is a bug.
@irexyc Thank you! Currently, do you know which APIs support batch inputs?
@nzomi
LMdeploy openai api doesn't support batch inputs. But if you send multiple requests in a short period of time, the requests will run in batch which will improve throughput.
好像是因为0.5.0版本有些默认参数和0.4.2不一致,你试试启动命令加这几个参数看看
--model-name internvl-internlm2 --session-len 32776 --num-tokens-per-iter 8192
也支持internvl2 我试过8B和26B-AWQ都能用
我用0.5.0版本也遇到一样的问题,运行InternVL2-26B两小时左右服务端卡住。我也试一下这个方法,看好使不。
Checklist
Describe the bug
使用0.5.0版本部署InternVL-Chat-V1-5-AWQ,python客户端使用openai方式发送请求后,服务端显示收到消息,但GPU的消耗资源无任何变化,客户端一直处于等待响应状态。
Reproduction
显卡信息: 4张魔改版2080ti 22G 部署代码: export CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server /data2/model_zoo/InternVL-Chat-V1-5-AWQ --backend turbomind --model-format awq --tp 4 --server-name 0.0.0.0 --server-port 8086 客户端访问代码:
python
def send_request(prompt, image_path, temperature, top_p, stream = True): base64_image = encode_image(image_path) response = client.chat.completions.create( model=model_name, messages=[{ 'role': 'user', 'content': [{ 'type': 'text', 'text': prompt, }, { 'type': 'image_url', 'image_url': { 'url': f"data:image/jpeg;base64,{base64_image}", }, }], }], temperature=temperature, top_p=top_p, max_tokens=1024, stream=stream,
frequency_penalty=0.5,
Environment
Error traceback