modelscope / ms-swift

Use PEFT or Full-parameter to finetune 400+ LLMs or 100+ MLLMs. (LLM: Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, Baichuan2, DeepSeek, Gemma2, ...; MLLM: Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL, Phi3.5-Vision, ...)
https://swift.readthedocs.io/zh-cn/latest/Instruction/index.html
Apache License 2.0
4.16k stars 368 forks source link

【新增功能需求】 Internvl2模型+VLLM 后端实现 异步客户端请求的Video 推理功能 #1921

Open PancakeAwesome opened 2 months ago

PancakeAwesome commented 2 months ago

Describe the feature Please describe the feature requested here(请在这里描述需求) Internvl2模型+VLLM 后端实现 异步客户端请求的Video 推理功能 Paste any useful information Paste any useful information, including papers, github links, etc.(请在这里描述其他有用的信息,比如相关的论文地址,github链接等) 现在的 vllm 2.4.0dev 分支运行以下 异步客户端请求逻辑会出错。 swift deploy --infer_backend vllm --max_model_len 2048 --host 127.0.0.1 --port 9122 --model_type internvl2-8b

from swift.llm import inference_client_async
question = '描述这个视频'
query = '<video>' + question
videos = ['./1.mp4']
tasks = [inference_client_async(model_type, query, videos=videos, request_config=request_config, port='9122') for _ in range(1)]
resp_list = asyncio.run(_batch_run(tasks))

>>>>>>>>>>>>>>>ERROR
ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/plain; charset=utf-8', url=URL('http://127.0.0.1:9122/v1/chat/completions')

Additional context Add any other context or information here(其他信息可以写在这里) ms-swift 2.4.0dev vllm 0.5.4

PancakeAwesome commented 2 months ago

@Jintao-Huang Looking forward to your reply, Thank u!