search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.11k stars 5.6k forks source link

[Core] Missing :authority header #40575

Open dioptre opened 11 months ago

dioptre commented 11 months ago

What happened + What you expected to happen

[2023-10-23 10:22:44,732: WARNING/MainProcess] Exception in thread [2023-10-23 10:22:44,733: WARNING/MainProcess] ray_print_logs [2023-10-23 10:22:44,733: WARNING/MainProcess] : [2023-10-23 10:22:44,733: WARNING/MainProcess] Traceback (most recent call last): [2023-10-23 10:22:44,733: WARNING/MainProcess] File "/Users/andrewgrosser/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [2023-10-23 10:22:44,735: WARNING/MainProcess]
[2023-10-23 10:22:44,735: WARNING/MainProcess] self.run() [2023-10-23 10:22:44,735: WARNING/MainProcess] File "/Users/andrewgrosser/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953, in run [2023-10-23 10:22:44,735: WARNING/MainProcess]
[2023-10-23 10:22:44,735: WARNING/MainProcess] self._target(*self._args, *self._kwargs) [2023-10-23 10:22:44,736: WARNING/MainProcess] File "/Users/andrewgrosser/Library/Caches/pypoetry/virtualenvs/ml-Yqfv2jYI-py3.10/lib/python3.10/site-packages/ray/_private/worker.py", line 804, in print_logs [2023-10-23 10:22:44,736: WARNING/MainProcess]
[2023-10-23 10:22:44,737: WARNING/MainProcess] data = subscriber.poll() [2023-10-23 10:22:44,737: WARNING/MainProcess] File "python/ray/_raylet.pyx", line 2840, in ray._raylet.GcsLogSubscriber.poll [2023-10-23 10:22:44,737: WARNING/MainProcess] File "python/ray/_raylet.pyx", line 457, in ray._raylet.check_status [2023-10-23 10:22:44,737: WARNING/MainProcess] ray.exceptions [2023-10-23 10:22:44,737: WARNING/MainProcess] . [2023-10-23 10:22:44,737: WARNING/MainProcess] RaySystemError [2023-10-23 10:22:44,737: WARNING/MainProcess] : [2023-10-23 10:22:44,737: WARNING/MainProcess] System error: Missing :authority header [2023-10-23 10:31:58,228: WARNING/MainProcess] Exception in thread [2023-10-23 10:31:58,229: WARNING/MainProcess] ray_listen_error_messages [2023-10-23 10:31:58,229: WARNING/MainProcess] : [2023-10-23 10:31:58,229: WARNING/MainProcess] Traceback (most recent call last): [2023-10-23 10:31:58,229: WARNING/MainProcess] File "/Users/andrewgrosser/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [2023-10-23 10:31:58,232: WARNING/MainProcess]
[2023-10-23 10:31:58,232: WARNING/MainProcess] self.run() [2023-10-23 10:31:58,232: WARNING/MainProcess] File "/Users/andrewgrosser/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953, in run [2023-10-23 10:31:58,232: WARNING/MainProcess]
[2023-10-23 10:31:58,233: WARNING/MainProcess] self._target(self._args, **self._kwargs) [2023-10-23 10:31:58,233: WARNING/MainProcess] File "/Users/andrewgrosser/Library/Caches/pypoetry/virtualenvs/ml-Yqfv2jYI-py3.10/lib/python3.10/site-packages/ray/_private/worker.py", line 2043, in listen_errormessages [2023-10-23 10:31:58,234: WARNING/MainProcess]
[2023-10-23 10:31:58,234: WARNING/MainProcess] , error_data = worker.gcs_error_subscriber.poll() [2023-10-23 10:31:58,234: WARNING/MainProcess] File "python/ray/_raylet.pyx", line 2795, in ray._raylet.GcsErrorSubscriber.poll [2023-10-23 10:31:58,234: WARNING/MainProcess] File "python/ray/_raylet.pyx", line 457, in ray._raylet.check_status [2023-10-23 10:31:58,235: WARNING/MainProcess] ray.exceptions [2023-10-23 10:31:58,235: WARNING/MainProcess] . [2023-10-23 10:31:58,235: WARNING/MainProcess] RaySystemError [2023-10-23 10:31:58,235: WARNING/MainProcess] : [2023-10-23 10:31:58,235: WARNING/MainProcess] System error: Missing :authority header

<function RemoteFunction._remote at 0x15e1e8550>
  File "/Users/andrewgrosser/Library/Caches/pypoetry/virtualenvs/ml-Yqfv2jYI-py3.10/lib/python3.10/site-packages/ray/remote_function.py", line 301, in _remote
    worker.function_actor_manager.export(self)
    │      │                      │      └ <ray.remote_function.RemoteFunction object at 0x17ee48d90>
    │      │                      └ <function FunctionActorManager.export at 0x15e1e97e0>
    │      └ <ray._private.function_manager.FunctionActorManager object at 0x15e244b50>
    └ <ray._private.worker.Worker object at 0x15e244bb0>
  File "/Users/andrewgrosser/Library/Caches/pypoetry/virtualenvs/ml-Yqfv2jYI-py3.10/lib/python3.10/site-packages/ray/_private/function_manager.py", line 260, in export
    if self._worker.gcs_client.internal_kv_exists(key, KV_NAMESPACE_FUNCTION_TABLE):
       │    │       │          │                  │    └ b'fun'
       │    │       │          │                  └ b"RemoteFunction:01000000:\xfcy\xfb\x9d\xe5p\x06\x83\x00\xb4'\xe2\x0e\xfe\xf2E\x16\xb1\x83\x1c\xfb\x1bE^\xc3\xdb \xbc"
       │    │       │          └ <cyfunction GcsClient.internal_kv_exists at 0x15d97d220>
       │    │       └ <ray._raylet.GcsClient object at 0x2adaaa500>
       │    └ <ray._private.worker.Worker object at 0x15e244bb0>
       └ <ray._private.function_manager.FunctionActorManager object at 0x15e244b50>
  File "python/ray/_raylet.pyx", line 2367, in ray._raylet._auto_reconnect.wrapper
  File "python/ray/_raylet.pyx", line 2355, in ray._raylet._auto_reconnect.wrapper
  File "python/ray/_raylet.pyx", line 2533, in ray._raylet.GcsClient.internal_kv_exists
  File "python/ray/_raylet.pyx", line 455, in ray._raylet.check_status

ray.exceptions.RpcError: Missing :authority header

Versions / Dependencies

version = "2.7.1"

Reproduction script

Crashes non-deterministically

Issue Severity

High: It blocks me from completing my task.

jjyao commented 11 months ago

@dioptre do you have repro code that I can run to repro and debug it?

jjyao commented 11 months ago

Mark as P1 until we have a repro.

dioptre commented 10 months ago

This generally happens if I run:

PROXY_HEALTH_CHECK_TIMEOUT_S=100 RAY_SERVE_DEBUG_MODE=1 RAY_SERVE_PROXY_READY_CHECK_TIMEOUT_S=100 RAY_ROTATION_MAX_BYTES=52428800 RAY_RUNTIME_ENV_LOG_TO_DRIVER_ENABLED=1 RAY_ENABLE_RECORD_ACTOR_TASK_LOGGING=1 ray start --head --port=6373   --ray-debugger-external --dashboard-host 0.0.0.0

and do any execution in the ray server

Exception in thread ray_print_logs:
Traceback (most recent call last):
  File "/Users/andrewgrosser/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/andrewgrosser/.pyenv/versions/3.10.12/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/andrewgrosser/Library/Caches/pypoetry/virtualenvs/ml-Yqfv2jYI-py3.10/lib/python3.10/site-packages/ray/_private/worker.py", line 804, in print_logs
    data = subscriber.poll()
  File "python/ray/_raylet.pyx", line 2840, in ray._raylet.GcsLogSubscriber.poll
  File "python/ray/_raylet.pyx", line 457, in ray._raylet.check_status
ray.exceptions.RaySystemError: System error: Missing :authority header
dioptre commented 10 months ago

Seems the startup script doesnt matter even if I write:

ray start --head --port=6373 --dashboard-host 0.0.0.0

Nightly build as of 11/21 also fails

dioptre commented 10 months ago

Is there a way just to turn this off?

dioptre commented 10 months ago

Maybe related https://github.com/nghttp2/nghttp2/issues/148

https://github.com/ray-project/ray/issues/11285

Looks like its used on mac at least

dioptre commented 10 months ago

Seems it effects Linux and x86_64 too.

log_to_driver=False was the workaround

🤯🔫

anyscalesam commented 7 months ago

the fact that the application crashes without a flag set is worrisome to me; we should at least discuss whether this is a usability versus stability issue and prioritize accordingly

aijdsofttech commented 5 months ago

I Got the same error and i posted this error on ray discussion forum Link : Error

asynchat commented 4 months ago

In FastAPI

ray.init()
ray.get(test.remote()) is ok.

@app.get("/")
async def root():
    ray.get(test.remote()) failure: ray.exceptions.RaySystemError: System error: Missing :authority header
    return {"message": "Hello World"}

Exception in thread ray_listen_error_messages: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045, in _bootstrap_inner self.run() File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 982, in run self._target(*self._args, **self._kwargs) File "/.venv/lib/python3.11/site-packages/ray/_private/worker.py", line 2136, in listen_errormessages , error_data = worker.gcs_error_subscriber.poll() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "python/ray/_raylet.pyx", line 3131, in ray._raylet.GcsErrorSubscriber.poll File "python/ray/_raylet.pyx", line 590, in ray._raylet.check_status ray.exceptions.RaySystemError: System error: Missing :authority header