[Feature Request] Compatibility of generated Pydantic models with FastAPI for proper OpenAPI Swagger serialization

[philipk19238]

Is your feature request related to a problem? Please describe.

I'm currently using betterproto to generate Pydantic models for my FastAPI application. However, FastAPI is unable to properly serialize the object to display it in the OpenAPI Swagger UI. This makes it difficult to understand and interact with the generated API documentation.

Describe the solution you'd like

I would like betterproto to generate Pydantic models that are compatible with FastAPI, allowing FastAPI to serialize the objects correctly in the OpenAPI Swagger UI. This could involve making any necessary modifications to the generated models, such as adding the appropriate Pydantic schema configuration or validators.

If anyone has any suggestions or ideas on how to implement this feature, I'd be happy to work on it myself and contribute to the project. This feature would greatly improve the user experience for developers using both betterproto and FastAPI, as it would allow for seamless integration of the two technologies.

[Gobot1234]

What about it doesn't work currently?

[philipk19238]

The serialization does not work properly for nested objects. For example:

Proto File

message Metadata {
    google.protobuf.Timestamp time_created = 1;
}

message OAuth {
    string access_token = 1;
    Metadata metadata = 2;
}

Endpoint

@app.get("/oauth")
def oauth(self, code) -> OAuth:
    ....
    model = OAuth(...)
    return model

Building the endpoint will cause the swagger page to display an internal server error with the following stack trace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 231, in openapi
    return JSONResponse(self.openapi())
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 206, in openapi
    self.openapi_schema = get_openapi(
  File "/usr/local/lib/python3.9/site-packages/fastapi/openapi/utils.py", line 423, in get_openapi
    definitions = get_model_definitions(
  File "/usr/local/lib/python3.9/site-packages/fastapi/utils.py", line 44, in get_model_definitions
    m_schema, m_definitions, m_nested_models = model_process_schema(
  File "/usr/local/lib/python3.9/site-packages/pydantic/schema.py", line 582, in model_process_schema
    m_schema, m_definitions, nested_models = model_type_schema(
  File "/usr/local/lib/python3.9/site-packages/pydantic/schema.py", line 623, in model_type_schema
    f_schema, f_definitions, f_nested_models = field_schema(
  File "/usr/local/lib/python3.9/site-packages/pydantic/schema.py", line 249, in field_schema
    s, schema_overrides = get_field_info_schema(field)
  File "/usr/local/lib/python3.9/site-packages/pydantic/schema.py", line 217, in get_field_info_schema
  schema_['default'] = encode_default(field.default)
  File "/usr/local/lib/python3.9/site-packages/pydantic/schema.py", line 996, in encode_default
  return pydantic_encoder(dft)
  File "/usr/local/lib/python3.9/site-packages/pydantic/json.py", line 90, in pydantic_encoder
  raise TypeError(f"Object of type '{obj.class.name}' is not JSON serializable")
  TypeError: Object of type 'object' is not JSON serializable

I'm not quite sure what is going on underneath the hood but my guess is that nested pydantic objects generated by betterproto are not "prerendered" and are instead placeholder objects, which causes an error when trying to serialize the annotations to JSON.

[Gobot1234]

I'm not quite sure what is going on underneath the hood but my guess is that nested pydantic objects generated by betterproto are not "prerendered" and are instead placeholder objects, which causes an error when trying to serialize the annotations to JSON.

Yeah that's pretty much it, not sure how you'd get around this safely

[philipk19238]

I think the best way may be to go through Pydantic and create a custom encoder for betterproto. I'll dig more into it when my time frees up.

[mikedh]

This would be awesome for utilities interacting with complicated proto messages! For what it's worth I hacked this for a demo using pydantic == 1.10.7 by patching a no-op encoder, which seemed to give better results than I was expecting. I didn't go through the whole message for correctness but it did appear to have generated the schema for the fields represented by the placeholder:

from fastapi import FastAPI
from pydantic import BaseModel, json

# use your own message built with betterproto here
from kerfed.protos.common.v1 import PartFabrication

# patch the pydantic encoders list in-place with a no-op for placeholders
from betterproto import _PLACEHOLDER
def encode_placeholder(obj: _PLACEHOLDER) -> dict:
    return {}
json.ENCODERS_BY_TYPE[_PLACEHOLDER] = encode_placeholder

app = FastAPI()

@app.post("/items/")
async def create_item() -> PartFabrication:
    return item

if __name__ == '__main__':
    # this will reproduce the crash
    a = app.openapi()

I agree the right place to PR this is probably pydantic, although I won't be able to.

[MarekPikula]

I'm using patched betterproto with patched flask-pydantic with success. The only catch is that I'm using pydantic dataclasses instead of BaseModels, but I think FastAPI should be able to handle it just fine (as per docs). The only requirement is to use the new --python_betterproto_opt=pydantic_dataclasses exporter argument and betterproto with this patch: #476. flask-pydantic changes are not upstreamed yet (available on my fork), but I think that since FastAPI has first-class support for Pydantic it's irrelevant for your use-case.

It would be nice if you could test this patch with FastAPI to see if my modification works in more scenarios (and maybe merge it sooner :stuck_out_tongue:).

[ValeriyMenshikov]

Don't work with fastAPI==0.103.1 , pydantic==1.10.12 ((

  m_schema, m_definitions, m_nested_models = model_process_schema(
  File "pydantic/schema.py", line 582, in pydantic.schema.model_process_schema
  File "pydantic/schema.py", line 623, in pydantic.schema.model_type_schema
  File "pydantic/schema.py", line 249, in pydantic.schema.field_schema
  File "pydantic/schema.py", line 217, in pydantic.schema.get_field_info_schema
  File "pydantic/schema.py", line 996, in pydantic.schema.encode_default
  File "pydantic/json.py", line 90, in pydantic.json.pydantic_encoder
TypeError: Object of type 'object' is not JSON serializable