vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
30.06k stars 4.54k forks source link

[Bug]: use `openai_vision_api_client.py` get error #7393

Closed jaffe-fly closed 3 months ago

jaffe-fly commented 3 months ago

Your current environment

The output of `python collect_env.py` ```text Your output of `python collect_env.py` here ```

🐛 Describe the bug

use vllm serve /mnt/models --trust-remote-code --load-format safetensors --tensor-parallel-size 2 --chat-template template_llava.jinjadeploy llava-1.5-7b-hfmodel.

vllm server log is

INFO 08-10 11:59:39 launcher.py:14] Available routes are:
INFO 08-10 11:59:39 launcher.py:22] Route: /openapi.json, Methods: GET, HEAD
INFO 08-10 11:59:39 launcher.py:22] Route: /docs, Methods: GET, HEAD
INFO 08-10 11:59:39 launcher.py:22] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 08-10 11:59:39 launcher.py:22] Route: /redoc, Methods: GET, HEAD
INFO 08-10 11:59:39 launcher.py:22] Route: /health, Methods: GET
INFO 08-10 11:59:39 launcher.py:22] Route: /tokenize, Methods: POST
INFO 08-10 11:59:39 launcher.py:22] Route: /detokenize, Methods: POST
INFO 08-10 11:59:39 launcher.py:22] Route: /v1/models, Methods: GET
INFO 08-10 11:59:39 launcher.py:22] Route: /version, Methods: GET
INFO 08-10 11:59:39 launcher.py:22] Route: /v1/chat/completions, Methods: POST
INFO 08-10 11:59:39 launcher.py:22] Route: /v1/completions, Methods: POST
INFO 08-10 11:59:39 launcher.py:22] Route: /v1/embeddings, Methods: POST
INFO:     Started server process [335]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8081 (Press CTRL+C to quit)

and use openai_vision_api_client.py in https://github.com/vllm-project/vllm/blob/main/examples/openai_vision_api_client.py test vllm,get this error

Error code: 400 - {'object': 'error', 'message': '', 'type': 'BadRequestError', 'param': None, 'code': 400}
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 466, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 652, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 805, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 490, in _make_request
    raise new_e
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='upload.wikimedia.org', port=443): Max retries exceeded with url: /wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/vllm-workspace/test.py", line 66, in <module>
    image_base64 = encode_image_base64_from_url(image_url=image_url)
  File "/vllm-workspace/test.py", line 59, in encode_image_base64_from_url
    with requests.get(image_url) as response:
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='upload.wikimedia.org', port=443): Max retries exceeded with url: /wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))

so whats wrong with it?

Isotr0py commented 3 months ago

Seems that accessing the example image on wikimedia failed due to network issue. Maybe you need to have a check on your network settings.