[BUG] serve.run host / port are deprecated, Cannot access port 8000 from outside the resource.

[JGSweets]

Ray has deprecated the usage of host and port is deprecated in ray since 2.7.0 https://docs.ray.io/en/releases-2.7.0/serve/api/doc/ray.serve.run.html

As a result, running remotely does not allow access to the service. More specifically, this line is not functioning as intended: https://github.com/ray-project/ray-llm/blob/master/rayllm/backend/server/run.py#L219

They suggest updating to use ray.serve.start to control host and port.

Thanks!

[MuhammadHadiofficial]

@JGSweets Faced same issues when tried running locally. Were you able to run by following the local running guide?

[sihanwang41]

@JGSweets @MuhammadHadiofficial , can you shows me what error message do you get? AFAIK The port arg is still functioning. we will update it in the future version of ray-llm.

[JGSweets]

@sihanwang41 @MuhammadHadiofficial I was able to manually fix this by using the -- rayllm run --model ... functionality and avoiding serve run ... command and altering rayllm/backend/server/run.py's def run:

host = "0.0.0.0"
serve._private.constants.DEFAULT_HTTP_HOST = host

This is a temporary fix and non-functional for deployments.

Instead, there needs to be an update to run which utilizes serve start and then deploys the application.

Inside ray's run, default is being used instead of the host parameter passed. Hence, it alters it to only using 127.0.0.1 Without a reverse proxy, there's no listener on incoming HTTP requests for the model serve port. Hence, one receives a 502 if they are trying to query the system externally.

[sihanwang41]

Hi @JGSweets , the serve run command is not using def run function. You can use serve run --host "0.0.0.0" xxxx to start the application by exposing the host publicly.

[JGSweets]

Correct, however, rayllm run was which allowed me to fix it in code and utilize it instead of serve run.

I can look into serve run --host ... at a later time and validate functionality. However, I'm concerned it won't have impact suggested as rayllm run replicated that functionality and did not work without the change because of the deprecation.

[sihanwang41]

Just did a verification, using the rayllm run without changing the code, it is using 0.0.0.0:

{"levelname": "INFO", "asctime": "2024-01-22 11:03:01,501", "component_name": "controller", "component_id": "7785", "message": "proxy_state.py:154 - Starting proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-ea8ce8aa4fc5efd64913f4b2279f6cf5789cbb69dc9ed749990b43ed' on node 'ea8ce8aa4fc5efd64913f4b2279f6cf5789cbb69dc9ed749990b43ed' listening on '0.0.0.0:8000'"}

You can see the log inside /tmp/ray/session_latest/logs/serve/controller_xxxx.log

Ray Serve still maintains the functionality with deprecation warning.

[JGSweets]

In that case, I'd expect the serve run to maintain the functionality as well. We might want to add a note to the README.md to use the arg for public deployments.

[JGSweets]

Confirmed that this works, thanks!