[serve][python3.11] Serve quick start example errors

[rickyyx]

What happened + What you expected to happen

Running serve quick start example from the ray website https://docs.ray.io/en/latest/ray-overview/index.html#ray-ai-runtime-quick-start errors:

RayTaskError(TypeError): ray::HTTPProxyActor.ready() (pid=3483137, ip=172.31.47.143, repr=<ray.serve._private.http_proxy.HTTPProxyActor object at 0x7f58b844fc10>)
  File "/data/home/rickyx/anaconda3/envs/dev-py311/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/home/rickyx/anaconda3/envs/dev-py311/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/home/rickyx/anaconda3/envs/dev-py311/lib/python3.11/site-packages/ray/serve/_private/http_proxy.py", line 424, in ready
    done_set, _ = await asyncio.wait(
                  ^^^^^^^^^^^^^^^^^^^
  File "/data/home/rickyx/anaconda3/envs/dev-py311/lib/python3.11/asyncio/tasks.py", line 415, in wait
    raise TypeError("Passing coroutines is forbidden, use tasks explicitly.")
TypeError: Passing coroutines is forbidden, use tasks explicitly.

Versions / Dependencies

wget e.g. https://buildkite.com/ray-project/oss-ci-build-pr/builds/8313#01855a61-6528-42d1-b55f-ae0e48ce236a pip install -U ray-3.0.0.dev0-cp311-cp311-manylinux2014_x86_64.whl

Reproduction script

Ray serve example from quick start

import requests
from starlette.requests import Request
from typing import Dict

from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier

from ray import serve

# Train model.
iris_dataset = load_iris()
model = GradientBoostingClassifier()
model.fit(iris_dataset["data"], iris_dataset["target"])

@serve.deployment(route_prefix="/iris")
class BoostingModel:
    def __init__(self, model):
        self.model = model
        self.label_list = iris_dataset["target_names"].tolist()

    async def __call__(self, request: Request) -> Dict:
        payload = (await request.json())["vector"]
        print(f"Received http request with data {payload}")

        prediction = self.model.predict([payload])[0]
        human_name = self.label_list[prediction]
        return {"result": human_name}

# Deploy model.
serve.run(BoostingModel.bind(model))

# Query it!
sample_request_input = {"vector": [1.2, 1.0, 1.1, 0.9]}
response = requests.get(
    "http://localhost:8000/iris", json=sample_request_input)
print(response.text)

Issue Severity

None

[rickyyx]

@zcin Guess this will be ray core's job to make sure this code works. Do you want to assign this to me instead?

[zcin]

@rickyyx Sure. The line directly causing the traceback you listed is not ray core related, though, so https://github.com/ray-project/ray/pull/31608 will fix that

[rickyyx]

Isn't the root cause ray core retruning corourintes as object references which are not compatible with async.wait? Or is there something else I am missing?

Oh I see, the ray core issue is tracked here: https://github.com/ray-project/ray/issues/31606

[zcin]

Isn't the root cause ray core retruning corourintes as object references which are not compatible with async.wait? Or is there something else I am missing?

The problem fixed in https://github.com/ray-project/ray/pull/31608 is passing coroutines to asyncio.wait. The ray core issue is passing object refs to asyncio.wait, and after https://github.com/ray-project/ray/pull/31608 is applied, running the Serve quickstart will show the ray core issue I think.

[ersinakinci]

Ray Serve quickstart is still broken. Can someone fix the docs/explain what you're supposed to do to get Ray Serve working? Thanks!

[Edrusb]

Hi, I hit to the same problem with latest package from pip (ray version 2.7.0) and python (version 3.11.4): ray.serve.start(), ray.serve.run() run OK, and have visible/expected counterpart on the dashboard, but when trying to obtain prediction, the model fails with:

> (HTTPProxyActor pid=791) ERROR 2023-10-02 09:47:43,092 http_proxy 172.17.0.3 85adb9a7-94cf-4591-b690-8cb12b6971f9 /mydemo mydemo http_proxy.py:1355 - Passing coroutines is forbidden, use tasks explicitly.
> (HTTPProxyActor pid=791) Traceback (most recent call last):
> (HTTPProxyActor pid=791)   File "/opt/conda/lib/python3.11/site-packages/ray/serve/_private/http_proxy.py", line 1342, in send_request_to_replica_streaming
> (HTTPProxyActor pid=791)     status_code = await self._consume_and_send_asgi_message_generator(
> (HTTPProxyActor pid=791)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> (HTTPProxyActor pid=791)   File "/opt/conda/lib/python3.11/site-packages/ray/serve/_private/http_proxy.py", line 1228, in _consume_and_send_asgi_message_generator
> (HTTPProxyActor pid=791)     obj_ref = next_obj_ref_task.result()
> (HTTPProxyActor pid=791)               ^^^^^^^^^^^^^^^^^^^^^^^^^^
> (HTTPProxyActor pid=791)   File "python/ray/_raylet.pyx", line 387, in _next_async
> (HTTPProxyActor pid=791)   File "/opt/conda/lib/python3.11/asyncio/tasks.py", line 415, in wait
> (HTTPProxyActor pid=791)     raise TypeError("Passing coroutines is forbidden, use tasks explicitly.")
> (HTTPProxyActor pid=791) TypeError: Passing coroutines is forbidden, use tasks explicitly.
> (HTTPProxyActor pid=791) ERROR:    ASGI callable returned without starting response.
> (ServeReplica:mydemo:MyModelDeployment pid=843) INFO 2023-10-02 09:47:43,099 MyModelDeployment mydemo#MyModelDeployment#DjSeNn 85adb9a7-94cf-4591-b690-8cb12b6971f9 /mydemo mydemo replica.py:749 - __CALL__ OK 0.2ms

[Edrusb]

Just seen that "Python 3.11 support is experimental." I guess the default pip package is not for python 3.11 (which comes by default with Debian, and similar distros for some time now). will try other distros with older python version... (python 3.10 seems supported). Fix me if I'm wrong, thanks!