search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
30.66k stars 4.65k forks source link

[Doc]: Docker+vllm+fastchat deploys multimodal large model Qwen2-vl-7b-instruct(docker+vllm+fastchat部署多模态大模型Qwen2-vl-7b-instruct) #10566

Open Aanlifire opened 11 hours ago

Aanlifire commented 11 hours ago

📚 The doc issue

When testing using the following method, an error is reported, and the terminal log results are shown as follows:

curl -X POST http:(base) root@lxing:~# curl -X POST http://127.0.0.1:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen2-VL-7B-Instruct", "messages": [{"role": "user", "content": "你好！"}], "temperature": 1.0, "max_tokens": 100 }' {"object":"error","message":"Internal Server Error","code":50001}

2024-11-22 02:50:27 | INFO | stdout | INFO: 127.0.0.1:55822 - "POST /model_details HTTP/1.1" 200 OK 2024-11-22 02:50:27 | INFO | stdout | INFO: 127.0.0.1:55838 - "POST /count_token HTTP/1.1" 200 OK 2024-11-22 02:50:27 | INFO | stdout | INFO: 127.0.0.1:55844 - "POST /worker_generate HTTP/1.1" 500 Internal Server Error 2024-11-22 02:50:27 | ERROR | stderr | ERROR: Exception in ASGI application 2024-11-22 02:50:27 | ERROR | stderr | Traceback (most recent call last): 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi 2024-11-22 02:50:27 | ERROR | stderr | result = await app( # type: ignore[func-returns-value] 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call 2024-11-22 02:50:27 | ERROR | stderr | return await self.app(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call 2024-11-22 02:50:27 | ERROR | stderr | await super().call(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/applications.py", line 113, in call 2024-11-22 02:50:27 | ERROR | stderr | await self.middleware_stack(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call 2024-11-22 02:50:27 | ERROR | stderr | raise exc 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call 2024-11-22 02:50:27 | ERROR | stderr | await self.app(scope, receive, _send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call 2024-11-22 02:50:27 | ERROR | stderr | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-11-22 02:50:27 | ERROR | stderr | raise exc 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app 2024-11-22 02:50:27 | ERROR | stderr | await app(scope, receive, sender) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 715, in call 2024-11-22 02:50:27 | ERROR | stderr | await self.middleware_stack(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 735, in app 2024-11-22 02:50:27 | ERROR | stderr | await route.handle(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle 2024-11-22 02:50:27 | ERROR | stderr | await self.app(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 76, in app 2024-11-22 02:50:27 | ERROR | stderr | await wrap_app_handling_exceptions(app, request)(scope, receive, send) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-11-22 02:50:27 | ERROR | stderr | raise exc 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app 2024-11-22 02:50:27 | ERROR | stderr | await app(scope, receive, sender) 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 73, in app 2024-11-22 02:50:27 | ERROR | stderr | response = await f(request) 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app 2024-11-22 02:50:27 | ERROR | stderr | raw_response = await run_endpoint_function( 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function 2024-11-22 02:50:27 | ERROR | stderr | return await dependant.call(**values) 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/fastchat/serve/vllm_worker.py", line 217, in api_generate 2024-11-22 02:50:27 | ERROR | stderr | output = await worker.generate(params) 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/fastchat/serve/vllm_worker.py", line 173, in generate 2024-11-22 02:50:27 | ERROR | stderr | async for x in self.generate_stream(params): 2024-11-22 02:50:27 | ERROR | stderr | File "/app/miniconda3/envs/vllm/lib/python3.12/site-packages/fastchat/serve/vllm_worker.py", line 106, in generate_stream 2024-11-22 02:50:27 | ERROR | stderr | sampling_params = SamplingParams( 2024-11-22 02:50:27 | ERROR | stderr | ^^^^^^^^^^^^^^^ 2024-11-22 02:50:27 | ERROR | stderr | TypeError: Unexpected keyword argument 'use_beam_search'

Suggest a potential alternative/fix

No response

Before submitting a new issue...