ray.services.get_node_ip_address doesn't work well if there is a local proxy

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

https://ray.io

Apache License 2.0

33.56k stars 5.7k forks source link

ray.services.get_node_ip_address doesn't work well if there is a local proxy #7316

Open tisonkun opened 4 years ago

tisonkun commented 4 years ago

https://github.com/ray-project/ray/blob/f14b6e477ba27b57805268741c3515524951fbf9/python/ray/services.py#L269

This method only works when there is no local proxy. Since Google DNS is not directly reachable everywhere it might detect a proxy address which is definitely not local address.

Is it possible we have a event better auto-detection?

dreamerlzl commented 4 years ago

Thank you for this issue. I have similar problems mentioned in the closed issue #6990; ray.init works only when my network is off (on macOS and I turn off the VPN and check the system preference>network to ensure there is no local proxy). When network is on, it says "[Errno 54] Connection reset by peer".

eric-tramel commented 4 years ago

See my followup comment in the discussion of #6990, but this is still a true issue that prevents the use of Ray on remote servers that are only reachable through VPN. In these cases, it is not possible to just disable local features and work locally. The VPN/proxy behavior is a required element of having a connection to a remote server where a ray instance/compute would be run.