Introduce a schema variant to reuse Validators, Serializers and CoreSchema

BoxyUwU commented 1 month ago

Introduces a nested CoreSchema variant that reuses the specified model's schema/validator/serializer. As an example, the core schema for the following pydantic models:

class A(BaseModel):
    field3: 'B'

class B(BaseModel):
    field1: list[A]

class Wrapper(BaseModel):
    a: A
    b: B

{
    'type': 'model',
    'cls': <class '__main__.A'>,
    'schema': {
        'type': 'model-fields',
        'fields': {
            'field3': {
                'type': 'model-field',
                'schema': {
                    'type': 'nested',
                    'cls': <class '__main__.B'>,
                    'get_info': <function GenerateSchema._generate_schema_from_property.<locals>.get_model_info at 0x7f9ec650be20>
                }
            }
        },
        'model_name': 'A'
    },
    'ref': '__main__.A:20628480'
}
{
    'type': 'model',
    'cls': <class '__main__.B'>,
    'schema': {
        'type': 'model-fields',
        'fields': {
            'field1': {
                'type': 'model-field',
                'schema': {
                    'type': 'list',
                    'items_schema': {
                        'type': 'nested',
                        'cls': <class '__main__.A'>,
                        'get_info': <function GenerateSchema._generate_schema_from_property.<locals>.get_model_info at 0x7f9ec650ba60>
                    }
                }
            }
        },
        'model_name': 'B'
    },
    'ref': '__main__.B:20632592'
}
{
    'type': 'model',
    'cls': <class '__main__.Wrapper'>,
    'schema': {
        'type': 'model-fields',
        'fields': {
            'a': {
                'type': 'model-field',
                'schema': {
                    'type': 'nested',
                    'cls': <class '__main__.A'>,
                    'get_info': <function GenerateSchema._generate_schema_from_property.<locals>.get_model_info at 0x7f9ec650bec0>
                }
            },
            'b': {
                'type': 'model-field',
                'schema': {
                    'type': 'nested',
                    'cls': <class '__main__.B'>,
                    'get_info': <function GenerateSchema._generate_schema_from_property.<locals>.get_model_info at 0x7f9ec65c8040>
                }
            }
        },
        'model_name': 'Wrapper'
    },
    'ref': '__main__.Wrapper:20635728'
}

Compare to the schema generated on main

``` { 'type': 'definitions', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.A:37460656'}, 'definitions': [ { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': {'field3': {'type': 'model-field', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.B:37433488'}}}, 'model_name': 'A' }, 'ref': '__main__.A:37460656' }, { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': {'field1': {'type': 'model-field', 'schema': {'type': 'list', 'items_schema': {'type': 'definition-ref', 'schema_ref': '__main__.A:37460656'}}}}, 'model_name': 'B' }, 'ref': '__main__.B:37433488' } ] } { 'type': 'definitions', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.B:37433488'}, 'definitions': [ { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': {'field3': {'type': 'model-field', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.B:37433488'}}}, 'model_name': 'A' }, 'ref': '__main__.A:37460656' }, { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': {'field1': {'type': 'model-field', 'schema': {'type': 'list', 'items_schema': {'type': 'definition-ref', 'schema_ref': '__main__.A:37460656'}}}}, 'model_name': 'B' }, 'ref': '__main__.B:37433488' } ] } { 'type': 'definitions', 'schema': { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': { 'a': {'type': 'model-field', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.A:37460656'}}, 'b': {'type': 'model-field', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.B:37433488'}} }, 'model_name': 'Wrapper' }, 'ref': '__main__.Wrapper:37440960' }, 'definitions': [ { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': {'field3': {'type': 'model-field', 'schema': {'type': 'definition-ref', 'schema_ref': '__main__.B:37433488'}}}, 'model_name': 'A' }, 'ref': '__main__.A:37460656' }, { 'type': 'model', 'cls': , 'schema': { 'type': 'model-fields', 'fields': {'field1': {'type': 'model-field', 'schema': {'type': 'list', 'items_schema': {'type': 'definition-ref', 'schema_ref': '__main__.A:37460656'}}}}, 'model_name': 'B' }, 'ref': '__main__.B:37433488' } ] } ```

This is similar to the current definitions-ref setup but it has a number of benefits:

The nested model does not need to have its schema built already in order to reuse it. Currently sometimes (I think) we build the core schema for a model twice if it has not been built by the first time it is referenced in another model.
The nested models schema is not inlined so the built CoreSchema is smaller which means walking it can be significantly faster in cases with lots of nested models when we do not need to recurse into `nested-model
There is no "flattening" step after generating the schema like with definitions schemas where we walk the built schema and ensure that any nested models' schemas does not have a definitions schema. This is what I believe is a significant portion of the performance wins since we now have to do significantly less complex tree traversals over python datastructures in python.
We can reuse validators and serializers with this setup which in theory should result in less memory overhead in cases with deeply recursive model definitions where the innermost model may have its validator constructed significantly more times than once

See pydantic/pydantic#10246 for the implementation of generating this new schema variant and also the benchmak results.

Before Merging

We don't propagate errors out of serde_serialize
The PyGcTraversible implementation for Result ignores the error variant as PyErr is not traversible
Make sure that the nested schema variant is defined flexibly enough to be useable for more than just models.
Make sure that pydantic/pydantic#10246 can be merged with positive perf wins otherwise there's no point :3

There is an issue pydantic/pydantic#10394 tracking future work on the pydantic side to use this schema more