Migrate dynamic schema for extra fields from drf-yasg

[valentijnscholten]

Not really a bug, but looking for the "best" (i.e. easiest) way to migrate a somewhat more complext construct from drf-yasg.

There is some existing code in this project that adds two Mixins to add a new parameter and a new response field to the list and read methods.

I am linking to the code as it's not so easy to isolate everything here.

Mixins: https://github.com/valentijnscholten/django-DefectDojo/blob/0dd151a860036465abed4868d5f9132470ce158e/dojo/api_v2/views.py#L333

Autoschema: https://github.com/valentijnscholten/django-DefectDojo/blob/0dd151a860036465abed4868d5f9132470ce158e/dojo/api_v2/views.py#L352

The idea is that all models there is a parameter prefetch that can be popluated with a comma seperated list of relationship names. So for a child, the parameter could contain parent. And then if a list of childs is being retrieved, the mixin will collect all the pk's of the parents. All these parent models are retrieved from the database and added to a new field prefetch in the response. The idea is that in one API call the children can be retrieved, including all relationships, in this example parents.

So the schema is dynamic as it depends on the relationships a model has. All the code for this with drf-yasg is in place and working, but it's quite a bit a boiler plat code with "Composable" schema's, Lazy references etc. So I am pondering what the easier way would be to achieve the same in drf-spectacular.

Anyone any thoughts? My first thought would be to write a post processor that just adds the parameter and response field + their schema's.

[valentijnscholten]

Managed to implement it using a PostProcessor: https://github.com/DefectDojo/django-DefectDojo/pull/4541/commits/97d47552e00bd8144ce2db710f8016534ad6a61a Maybe not the cleanest method, but quite portable as it almost doesn't use any drf-spectacular code.

[tfranzel]

glad you managed to get a solution. i had a look but the issue was involved and i was unable to come up with a good suggestion without investing too much time.

[valentijnscholten]

Turns out my solution is still missing a piece. I need a mapping from endpoint (path) to serializer, which the postprocessor doesn't have access to I think? I will try to dig deep into the generator and see if I can map something. Still feels I am doing something wrong, or we're missing a more straightforward way to hook into the schema generation.

[tfranzel]

the view schemas are accessible through result. that much you know. the serializer is accessible through the registry. you can access it with generator.registry. there you have all the ResolvedComponents available. if you change the registry remember to regenerate it at the end.

    result['components'] = generator.registry.build(spectacular_settings.APPEND_COMPONENTS)

have a look at hooks.py. the enum postprocessing uses pretty much all the tricks. not sure if that helps you further. generally the design was not accounting for wildly dynamic views. it could be possible that this tricky and there is no easier way. can't be sure though as i do not completely understand what you are doing.

[valentijnscholten]

I am trying to do more or less this:

1) Find all API views 2) For some API views (either subclass of a PrefetchListMixin or view having a parameter named 'prefetch') 3) Find the serializer for that view 4) Find all the fields for that serializer 5) Select only those fields that are a foreignkey or manyotmany relation 6) Find the request schema / component for the view 7) Add the list of these field names as enum to the schema of the prefetch parameter 8) Find the serializer for each of those fields 9) Find the component ref for that serializer 8) Find the response schema / component for the view 9) Add a dictionary to the response schema / component: https://swagger.io/docs/specification/data-models/dictionaries/ key = string, value = ref to component name

For example if a model has 2 relationships (members and product_manager), the prefetch parameter and prefetch responses will have this schema:

                    {
                        "in": "query",
                        "name": "prefetch",
                        "schema": {
                            "type": "array",
                            "items": {
                                "type": "string",
                                "enum": [
                                    "members",
                                    "product_manager",
                                ]
                            }
                        },
                        "description": "List of fields for which to prefetch model instances and add those to the response"
                    },

"responses": {
                    "200": {
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/PaginatedProductList"
                                }
                            }
                        },
                        "description": ""
                    }
                }

 "PaginatedProductList": {
                "type": "object",
                "properties": {
                    "count": {
                        "type": "integer",
                        "example": 123
                    },
                    "next": {
                        "type": "string",
                        "nullable": true,
                        "format": "uri",
                        "example": "http://api.example.org/accounts/?offset=400&limit=100"
                    },
                    "previous": {
                        "type": "string",
                        "nullable": true,
                        "format": "uri",
                        "example": "http://api.example.org/accounts/?offset=200&limit=100"
                    },
                    "results": {
                        "type": "array",
                        "items": {
                            "$ref": "#/components/schemas/Product"
                        }
                    },
                    "prefetch": {
                        "type": "object",
                        "properties": {
                            "members": {
                                "type": "object",
                                "readOnly": true,
                                "additionalProperties": {
                                    "$ref": "#/components/schemas/UserStub"
                                }
                            },
                            "product_manager": {
                                "type": "object",
                                "readOnly": true,
                                "additionalProperties": {
                                    "$ref": "#/components/schemas/UserStub"
                                }
                            },
                        }
                    }
                }
            },

In my current implementation I just went over all the paths in the result parameter in the post processor. But then I miss information to do step 3 as the result doesn't have the serializers/views. But maybe the generator has it somewhere, for example in _get_paths_and_endpoints? I looked at the registry but that just contains the components and not the views/parameters/serializers.