search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.24k stars 5.81k forks source link

[Core] --redis-password= NOT Working :( #48226

Open AzizDXT opened 1 month ago

AzizDXT commented 1 month ago

Ray password not working!

ray start --head --port=80 --node-ip-address=172.18.87.191 --dashboard-host=0.0.0.0 --redis-password='343124241578'

Result :

(venv) PS C:\Users\ADMIN> ray start --address='172.18.87.191:80'
Local node IP: 172.18.80.1

--------------------
Ray runtime started.
--------------------

To terminate the Ray runtime, run
  ray stop
(venv) PS C:\Users\ADMIN>

Ray program contacted the node without the password unfortunately

Versions / Dependencies

Package Version

absl-py 2.1.0 aiohappyeyeballs 2.4.3 aiohttp 3.10.10 aiohttp-cors 0.7.0 aiosignal 1.3.1 annotated-types 0.7.0 anyio 4.6.2.post1 astunparse 1.6.3 attrs 24.2.0 cachetools 5.5.0 certifi 2024.8.30 charset-normalizer 3.4.0 click 8.1.7 cloudpickle 3.1.0 colorful 0.5.6 cupy-cuda12x 13.3.0 Deprecated 1.2.14 distlib 0.3.9 dm-tree 0.1.8 Farama-Notifications 0.0.4 fastapi 0.115.3 fastrlock 0.8.2 filelock 3.16.1 flatbuffers 24.3.25 frozenlist 1.4.1 fsspec 2024.10.0 gast 0.6.0 google-api-core 2.21.0 google-auth 2.35.0 google-pasta 0.2.0 googleapis-common-protos 1.65.0 grpcio 1.67.0 gymnasium 0.28.1 h11 0.14.0 h5py 3.12.1 httptools 0.6.4 idna 3.10 imageio 2.36.0 importlib_metadata 8.4.0 jax-jumpy 1.0.0 Jinja2 3.1.4 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 keras 3.6.0 lazy_loader 0.4 libclang 18.1.1 linkify-it-py 2.0.3 lz4 4.3.3 Markdown 3.7 markdown-it-py 3.0.0 MarkupSafe 3.0.2 mdit-py-plugins 0.4.2 mdurl 0.1.2 memray 1.14.0 ml-dtypes 0.4.1 msgpack 1.1.0 multidict 6.1.0 namex 0.0.8 networkx 3.4.2 numpy 1.26.4 opencensus 0.11.4 opencensus-context 0.1.3 opentelemetry-api 1.27.0 opentelemetry-exporter-otlp 1.27.0 opentelemetry-exporter-otlp-proto-common 1.27.0 opentelemetry-exporter-otlp-proto-grpc 1.27.0 opentelemetry-exporter-otlp-proto-http 1.27.0 opentelemetry-proto 1.27.0 opentelemetry-sdk 1.27.0 opentelemetry-semantic-conventions 0.48b0 opt_einsum 3.4.0 optree 0.13.0 packaging 24.1 pandas 2.2.3 pillow 11.0.0 pip 24.2 platformdirs 4.3.6 prometheus_client 0.21.0 propcache 0.2.0 proto-plus 1.24.0 protobuf 4.25.5 psutil 6.1.0 py-spy 0.3.14 pyarrow 17.0.0 pyasn1 0.6.1 pyasn1_modules 0.4.1 pydantic 2.9.2 pydantic_core 2.23.4 Pygments 2.18.0 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 pytz 2024.2 PyYAML 6.0.2 ray 2.37.0 referencing 0.35.1 requests 2.32.3 rich 13.9.3 rpds-py 0.20.0 rsa 4.9 scikit-image 0.24.0 scipy 1.14.1 setuptools 75.2.0 shellingham 1.5.4 six 1.16.0 smart-open 7.0.5 sniffio 1.3.1 starlette 0.41.0 tensorboard 2.17.1 tensorboard-data-server 0.7.2 tensorboardX 2.6.2.2 tensorflow 2.17.0 termcolor 2.5.0 textual 0.84.0 tifffile 2024.9.20 typer 0.12.5 typing_extensions 4.12.2 tzdata 2024.2 uc-micro-py 1.0.3 urllib3 2.2.3 uvicorn 0.32.0 uvloop 0.21.0 virtualenv 20.27.0 watchfiles 0.24.0 websockets 13.1 Werkzeug 3.0.4 wheel 0.44.0 wrapt 1.16.0 yarl 1.16.0 zipp 3.20.2

Reproduction script

└─# ray start --head --port=80 --node-ip-address=172.18.87.191 --dashboard-host=0.0.0.0 --redis-password='12345678' Usage stats collection is disabled.

Local node IP: 172.18.87.191

Ray runtime started.

Next steps To add another node to this Ray cluster, run ray start --address='172.18.87.191:80'

To connect to this Ray cluster: import ray ray.init(_node_ip_address='172.18.87.191')

To submit a Ray job using the Ray Jobs CLI: RAY_ADDRESS='http://172.18.87.191:8265' ray job submit --working-dir . -- python my_script.py

See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html for more information on submitting Ray jobs to the Ray cluster.

To terminate the Ray runtime, run ray stop

To view the status of the cluster, use ray status

To monitor and debug Ray, view the dashboard at 172.18.87.191:8265

If connection to the dashboard fails, check your firewall settings and network configuration.

┌──(venv)(root㉿ADMIN)-[~]

Issue Severity

Medium: It is a significant difficulty but I can work around it.

rynewang commented 2 weeks ago

I think in order to use --redis-password you need to set RAY_REDIS_ADDRESS=xxx

AzizDXT commented 2 weeks ago

tell me how u do it?

jjyao commented 1 week ago

@AzizDXT you can do RAY_REDIS_ADDRESS=xxxx ray start --head --redis-password=xxxx

AzizDXT commented 1 week ago

@AzizDXT you can do RAY_REDIS_ADDRESS=xxxx ray start --head --redis-password=xxxx

(openet) PS C:\Users\Abdulaziz> $env:RAY_REDIS_ADDRESS="127.0.0.1:6379" (openet) PS C:\Users\Abdulaziz> ray start --head --redis-password=12345 --port=6379 Usage stats collection is disabled.

Local node IP: 127.0.0.1 [2024-11-19 05:54:57,769 C 39760 23644] (ray_init) redis_context.cc:487: Check failed: _s.ok() Bad status: RedisError: connection reset StackTrace Information unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown

[2024-11-19 05:54:57,769 C 39760 23644] (ray_init) redis_context.cc:487: Check failed: _s.ok() Bad status: RedisError: connection reset StackTrace Information unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown

(openet) PS C:\Users\Abdulaziz> (openet) PS C:\Users\Abdulaziz> redis-cli ping (openet) PS C:\Users\Abdulaziz> redis-server [36276] 19 Nov 05:56:06.131 # Warning: no config file specified, using the default config. In order to specify a config file use C:\Program Files\Redis\redis-server.exe /path/to/redis.conf [36276] 19 Nov 05:56:06.134 # Creating Server TCP listening socket *:6379: bind: No such file or directory (openet) PS C:\Users\Abdulaziz> netsh advfirewall firewall add rule name="Open Redis Port 6379" dir=in action=allow protocol=TCP localport=6379 Ok.

(openet) PS C:\Users\Abdulaziz> redis-cli ping Error: An existing connection was forcibly closed by the remote host. (openet) PS C:\Users\Abdulaziz> netstat -ano | findstr :6379 TCP 0.0.0.0:6379 0.0.0.0:0 LISTENING 5020 TCP [::]:6379 [::]:0 LISTENING 7328 (openet) PS C:\Users\Abdulaziz> taskkill /PID 5020 /F SUCCESS: The process with PID 5020 has been terminated. (openet) PS C:\Users\Abdulaziz> taskkill /PID 7328 /F SUCCESS: The process with PID 7328 has been terminated. (openet) PS C:\Users\Abdulaziz> ray start --head --redis-password=12345 --port=6379 Usage stats collection is disabled.

Local node IP: 127.0.0.1 [2024-11-19 05:58:04,490 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:04,490 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:07,008 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:07,008 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:09,530 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:09,530 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:12,052 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:12,052 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:14,569 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:14,569 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:17,101 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:17,101 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:19,619 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:19,619 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:22,143 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:22,143 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:24,668 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:24,668 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:27,208 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms. [2024-11-19 05:58:27,208 E 46544 37896] (ray_init) redis_context.cc:362: Failed to connect to Redis due to: RedisError: Could not establish connection to Redis 127.0.0.1:6379 (context.err = 1).. Will retry in 500ms.

Not working