[Bug] Ray Hydra Launcher fails to connect to the GCS

[stwerner97]

🐛 Bug

Description

Using the Hydra Ray launcher, I want to submit the Simple Ray Launcher Example of Hydra to a remote Ray Kubernetes cluster. Launching the job via python my_app.py --multirun hydra/launcher=ray hydra.launcher.ray.init.address=localhost:8265, however, fails to connect to the GCS of Ray.

[2023-07-11 16:21:28,604][HYDRA] Ray Launcher is launching 1 jobs, sweep output dir: multirun/2023-07-11/16-21-28
[2023-07-11 16:21:28,604][HYDRA] Initializing ray with config: {'address': 'localhost:8265'}
2023-07-11 16:21:28,604 INFO worker.py:1452 -- Connecting to existing Ray cluster at address: <ip-address>:8265...
2023-07-11 16:21:33,615 ERROR utils.py:1390 -- Failed to connect to GCS. Please check `gcs_server.out` for more details.
2023-07-11 16:21:33,615 WARNING utils.py:1397 -- Unable to connect to GCS at <ip-address>:8265. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access.
2023-07-11 16:21:40,629 ERROR utils.py:1390 -- Failed to connect to GCS. Please check `gcs_server.out` for more details.
2023-07-11 16:21:40,630 WARNING utils.py:1397 -- Unable to connect to GCS at <ip-address>:8265. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access.

I have configured the Ray launcher plugin to point towards the address of the (port-forwarded) Ray dashboard. I verified that I can successfully submit jobs using Ray, i.e., ray job submit --address http://localhost:8265 -- python script.py is successful.

Some information on the Ray Kubernetes cluster:

• The raycluster-kuberay-head uses the image rayproject/ray:2.3.0.
• The kuberay-operator uses the image kuberay/operator:v0.5.0.
• The raycluster-kuberay-head-svc service has the following targets
  • app.kubernetes.io/created-by=kuberay-operator: <ip-address>:10001 10001/TCP
  • app.kubernetes.io/name=kuberay: <ip-address>:6379 6379/TCP
  • ray.io/cluster=raycluster-kuberay: <ip-address>:8265 8265/TCP (this is the forwarded port)
  • ray.io/identifier=raycluster-kuberay-head: 8080/TCP
  • ray.io/node-type=head: <ip-address>:8000 8000/TCP

I didn't find too much information on this issue and am unsure whether this issue belongs to the Hydra or Ray repository.

Checklist

• [x] I checked on the latest version of Hydra
• [ ] I created a minimal repro (See this for tips).

System information

• Hydra Version : 1.3.2
• Python version : 3.7.16

[shagunsodhani]

I am not very familiar with the ray-launcher but I will try to take a stab here.

You mentioned the address http://localhost:8265 in the standalone command ray job submit --address http://localhost:8265 -- python script.py while you use hydra.launcher.ray.init.address=localhost:6379 in the config. Shouldn't these be the same ?

[stwerner97]

Hi @shagunsodhani , thanks for responding! 😊

Yes, they should be the same, I made a copy-paste error when reporting the issue. The issue also occurs for hydra.launcher.ray.init.address=localhost:8265. I've edited the initial issue report and corrected the port number.