Schema generation

[AdrianSosic]

Hi @Tinche! I really love the combination of attrs and cattrs; it is a great way to organize code! After having shot myself in the foot several times with pydantic, I switched to (c)attrs in most of my projects. However, there is one part where pydantic still has an edge – or at least I couldn't find an immediate solution with (c)attrs – which is generating schemas (JSON or similar) from class definitions.

Perhaps this hasn't been in scope so far, but I think this would be a natural task for cattrs, no? At least I know that other people seem interested in this as well. A quick search yields:

• https://news.ycombinator.com/item?id=28313047
• https://pypi.org/project/attrs-jsonschema/

Do you think this is something that should become part of the roadmap?

[Tinche]

Howdy,

this is definitely an interesting area to pursue. Let me first direct your attention to one of my related projects: https://uapi.threeofwands.com/en/latest/. uapi already has an OpenAPI schema generator built-in. It's not exactly built on cattrs, it just works in parallel with it.

I'm just not very happy with it. Right now it's not very customizable and it's very tied to uapi, and I want to figure out how to solve both problems first. There's also the question of where it should ultimately live, in uapi, a separate repo or here in cattrs?

Another issue is that this is an OpenAPI generator; you're enquiring about a subset (jsonschema). I'm not sure what a composable API would look like here.

I think architecturally I'm leaning towards the idea of this being a layer on top of cattrs. So instead of dealing with a cattrs Converter, you'd be dealing with a different class, internally wrapping a cattrs Converter (or two).

So it's fair to say this is on the horizon ;)

[AdrianSosic]

Thanks for sharing your thoughts. It's great news to hear that this feature might be coming at some point!

Could you perhaps elaborate on why you think you need yet another layer on top of cattrs? I'm asking because:

1. The converters give you exactly the composable API you are looking for.
2. Effectively, the converters are really what specify the schema – though rather implicitly at the moment. In a sense, a converter ultimately serves the purpose of a validator for a schema using a specific input, right? If it the input can be structured, it is in line with the schema; if not, it violates it. So "all" that is necessary (<-- simply speaking! I have no idea how complicated this really is in the end :D) is to turn these implicit rules into an explicit summary.

But maybe we are talking about the same idea here.

In the meantime, I would give uapi a try and see if it fits my purpose. Could you perhaps briefly clarify:

• If I have some (nested) attrs classes, how exactly do I get the schema for it? Is there somewhere a minimal example that I can reuse?
• Is there any way to specifically control the (de-)serialization mechanism? That is, if I have an existing set of nested attrs classes and defined custom cattrs hooks for it, is it possible to transfer that same logic to uapi and use it to extract the resulting schema?

[salotz]

I'll put my vote in for having a converter specifically for outputting JSONSchema, and not really OpenAPI.

Currently for my internal web "framework" I've built on top of Starlette I have my own way of writing and annotating endpoints with OpenAPI specific stuff (e.g. Endpoint classes with methods as operations, etc.). Then I do reflection on types for generating the JSONSchema components to plug into the OpenAPI I generate.

The outer OpenAPI stuff isn't too bad, but handling all the complex type reflection for the JSONSchema is really a PITA. Having a composable cattrs like tool would be really awesome and reusable across different web "frameworks".

I can't answer whether having the JSONSchema generation as a preconfigured converter or a different library would be better, but I think for that specifically a preconf module makes sense. Something like msgspec has.

If you want to handle the full OpenAPI generation part I would definitely do that as a separate (and also generic) module that can have specific support for cattrs (if that is even needed at all).

Then you can use this as a component of uapi or whatever and use the endpoint "sugar" and HTTP handling that you prefer and just feed that into the cattrs.preconf.jsonschema and OpenAPI module.

I already have actually a lot of this code which is fairly decoupled from the web framework side of things. Happy to contribute it if @Tinche can use his cattrs wizardry to help with the JSONSchema generation part :)

Here is a sample of the entrypoint for the whole OpenAPI schema:

from openapi_spec_validator import OpenAPIV31SpecValidator, validate
OPENAPI_SPEC_VALIDATOR = OpenAPIV31SpecValidator

@define
class OpenAPISchema:
    """Model for an entire OpenAPI specification."""

    openapi_version: str
    info: OpenAPISchemaInfo
    api_paths: list[OpenAPIPath]
    servers: list[OpenAPIServer] = Factory(list)
    components: OpenAPIComponents | None = None

    def validate(self) -> OpenAPIValidationError | None:
        spec_d = self.to_dict()
        try:
            validate(
                spec_d,  # type: ignore
                cls=OPENAPI_SPEC_VALIDATOR,
            )
        except OpenAPIValidationError as exc:
            return exc
        else:
            return None

    def to_dict(self) -> APISpecDict:
        return {
            "openapi": self.openapi_version,
            "info": spec_info_to_openapi(self.info),
            "paths": api_paths_to_openapi(self.api_paths),
            **(
                {"servers": servers_to_openapi(self.servers)}
                if len(self.servers) > 0
                else {}
            ),
            **(
                {"components": components_to_openapi(self.components)}
                if self.components is not None
                else {}
            ),
        }

    def to_json(self) -> str:
        return json.dumps(self.to_dict())

    def to_yaml(self) -> str:
        return to_yaml(self.to_dict())

To adapt to your web framework you need to generate OpenAPIComponents. Which for me is something like this:

from my_app.api import API

components: OpenAPIComponents = generate_component_schemas(API.routes)

openapi_schema = OpenAPISchema(components=components, ...)

openapi_schema.to_json()
openapi_schema.to_yaml()

[salotz]

I took a look at your uapi.openapi code. I can see how this is fairly reusable and has a lot of similar ideas.

Ideally I'd like to just remove most of the code I wrote for OpenAPI in my code base. We should put our heads together to get a solution. More than happy to test on this currently just thinking about ripping out my OpenAPI code since it just causes us headaches and its easier to just write it out by hand.