[Serve] FastAPI integration is not passing arguments correctly in the POST method

[vilsonrodrigues]

What happened + What you expected to happen

With the migration of Ray to the new version 2.7.x, I am testing using FastAPI.

Before the problem: every argument that comes via POST request is now being converted into ray.serve.api.Ingress object at 0x. This prevents me from unpacking the JSON data in the first function. This object does not have methods like await or work with ray.get(myobject).

The simple way out would be to convert from async def to def.

I discovered that if I use a FastAPI Form for each function parameter, I can receive the data correctly.

When I add a type validation that has two options like str | list[str], the function returns only the 1st value in the list. To work the entire list I have to specify just list[str]. If I specify that the parameter can be str | None, and not passing this parameter I get error.

If I create a Pydantic Class for a function parameter I start to receive an error:

{"detail":[{"loc":["body","prompt"],"msg":"field required","type":"value_error.missing"},{"loc":["body" ,"model"],"msg":"field required","type":"value_error.missing"}]}

Versions / Dependencies

Ray v2.7.1 Python 3.10 Ubuntu 22.04.2 LTS

fastapi v0.104.0 pydantic v1.10.13

Reproduction script

1. Multi string input. Error: value_error.missing

code

from fastapi import FastAPI
from ray import serve

app = FastAPI()

@serve.deployment
@serve.ingress(app)
class Ingress:    
    def __init__(self):
        pass

    @app.post("/model/")
    async def model(self, prompt: str, model: str):
        print(type(prompt))
        print(type(model))
        return 200

app = Ingress.bind()        

test

import requests

url = "http://localhost:8000/model/"

data = {
    "prompt": "my test",
    "model": "h(x)"
}

response = requests.post(url, data=data)

print(response.text)

error

{"detail":[{"loc":["query","prompt"],"msg":"field required","type":"value_error.missing"},{"loc":["query","model"],"msg":"field required","type":"value_error.missing"}]}

If I remove self from the function, the error remains

2. Success with One parameter. Obtain object ref. But if add self returns same error

code

from fastapi import FastAPI
from ray import serve

app = FastAPI()

@serve.deployment
@serve.ingress(app)
class Ingress:

    def __init__(self):
        pass

    @app.post("/model/")
    async def model(prompt: str):
        print(type(prompt))
        print(prompt)
        return 200

app = Ingress.bind()        

test

import requests

url = "http://localhost:8000/model/"

data = {
    "prompt": "my test",
}

response = requests.post(url, data=data)

print(response.text)

shell

(ServeReplica:default:Ingress pid=79252) <class 'ray.serve.api.Ingress'>
(ServeReplica:default:Ingress pid=79252) <ray.serve.api.Ingress object at 0x7f77d61cf7f0>

function with self

async def model(self, prompt: str):

error

{"detail":[{"loc":["query","prompt"],"msg":"field required","type":"value_error.missing"}]}

3. Form use, it works

code

from fastapi import FastAPI, Form
from ray import serve

app = FastAPI()

@serve.deployment
@serve.ingress(app)
class Ingress:

    def __init__(self):
        pass

    @app.post("/model/")
    async def model(self, prompt: str = Form(...), model: str = Form(...)):
        print(prompt)
        print(model)
        return 200

app = Ingress.bind()               

test

import requests

url = "http://localhost:8000/model/"

data = {
    "prompt": "my test",
    "model": "h(x)"
}

response = requests.post(url, data=data)

print(response.text)

shell

(ServeReplica:default:Ingress pid=84969) my test
(ServeReplica:default:Ingress pid=84969) h(x)

4. Error: Union hint

code

%%file objectref.py
from fastapi import FastAPI, Form
from ray import serve

app = FastAPI()

@serve.deployment
@serve.ingress(app)
class Ingress:

    def __init__(self):
        pass

    @app.post("/model/")
    async def model(self, prompt: str = Form(...), model: str | None = Form(...)):
        print(prompt)
        print(model)
        return 200

app = Ingress.bind()               

test

import requests

url = "http://localhost:8000/model/"

data = {
    "prompt": "my test",
}

response = requests.post(url, data=data)

print(response.text)

error

{"detail":[{"loc":["body","model"],"msg":"field required","type":"value_error.missing"}]}

using Union Typing operator does not change the error

async def model(self, prompt: str = Form(...), model: Union[str,None] = Form(...)):

5. If str or list str, get only last list value

code

from fastapi import FastAPI, Form
from ray import serve

app = FastAPI()

@serve.deployment
@serve.ingress(app)
class Ingress:

    def __init__(self):
        pass

    @app.post("/model/")
    async def model(self, prompt: str = Form(...), model: str | list[str] = Form(...)):
        print(prompt)
        print(model)
        return 200

app = Ingress.bind()               

test

import requests

url = "http://localhost:8000/model/"

data = {
    "prompt": "my test",
    "model": ["u(x)","dydx","bytes"]
}

response = requests.post(url, data=data)

print(response.text)

shell

(ServeReplica:default:Ingress pid=92011) my test
(ServeReplica:default:Ingress pid=92011) bytes

Issue Severity

High: It blocks me from completing my task.

[anyscalesam]

@sihanwang41 please triage

[akshay-anyscale]

@edoakes or @zcin is this fixed?