Support for schema with circular references/recursive data structures

[ashb]

Hello,

I'm trying to use Speakeasy to generate a client for a (modified/in-progress/draft) Apache Airflow API, and I want to make use of a recursive/circular data structure. To give you a minimal repro case, here is a short pair of pydantic models I want to create:

class Task(BaseModel):
    id: str | None
    label: str | None
    type: Literal["task"]
    is_mapped: bool | None
    operator: str
    extra_links: list[str]
    has_outlet_datasets: bool

class TaskGroup(BaseModel):
    id: str | None
    type: Literal["task_group"]
    label: str | None
    is_mapped: bool | None
    tooltip: str | None
    children: list[Task | TaskGroup]

This causes the app to show an error about circular references. Is this on your roadmap to allow?

[ndimares]

Hey @ashb, do you mind sending over the openapi schema you are working off of? I believe that we do support circular references in general, except in the case where a schema has required=true and is part of an unrecoverable infinite loop.

[ashb]

Let me make a reduced test case for you.

Here you go @ndimares

{
  "components": {
    "schemas": {
      "DagStructure": {
        "additionalProperties": false,
        "properties": {
          "edges": {
            "items": {
              "$ref": "#/components/schemas/Edge"
            },
            "type": "array"
          },
          "group": {
            "allOf": [
              {
                "$ref": "#/components/schemas/TaskGroup"
              }
            ]
          },
          "ordering": {
            "items": {
              "type": "string"
            },
            "type": "array"
          }
        },
        "required": [
          "group",
          "ordering",
          "edges"
        ],
        "title": "DagStructure",
        "type": "object"
      },
      "Edge": {
        "additionalProperties": false,
        "properties": {
          "from": {
            "title": "From",
            "type": "string"
          },
          "label": {
            "type": "string"
          },
          "to": {
            "type": "string"
          }
        },
        "required": [
          "from",
          "to"
        ],
        "title": "Edge",
        "type": "object"
      },
      "HTTPValidationError": {
        "properties": {
          "detail": {
            "items": {
              "$ref": "#/components/schemas/ValidationError"
            },
            "title": "Detail",
            "type": "array"
          }
        },
        "title": "HTTPValidationError",
        "type": "object"
      },
      "Task": {
        "additionalProperties": false,
        "properties": {
          "extraLinks": {
            "items": {
              "type": "string"
            },
            "title": "Extra Links",
            "type": "array"
          },
          "hasOutletDatasets": {
            "title": "Has Outlet Datasets",
            "type": "boolean"
          },
          "id": {
            "type": "string"
          },
          "isMapped": {
            "title": "Is Mapped",
            "type": "boolean"
          },
          "label": {
            "type": "string"
          },
          "operator": {
            "type": "string"
          },
          "type": {
            "enum": [
              "task"
            ],
            "type": "string"
          }
        },
        "required": [
          "type",
          "operator",
          "extraLinks",
          "hasOutletDatasets"
        ],
        "title": "Task",
        "type": "object"
      },
      "TaskGroup": {
        "additionalProperties": false,
        "properties": {
          "children": {
            "items": {
              "anyOf": [
                {
                  "$ref": "#/components/schemas/Task"
                },
                {
                  "$ref": "#/components/schemas/TaskGroup"
                }
              ]
            },
            "type": "array"
          },
          "id": {
            "type": "string"
          },
          "isMapped": {
            "title": "Is Mapped",
            "type": "boolean"
          },
          "label": {
            "type": "string"
          },
          "tooltip": {
            "type": "string"
          },
          "type": {
            "enum": [
              "task_group"
            ],
            "type": "string"
          }
        },
        "required": [
          "type",
          "children"
        ],
        "title": "TaskGroup",
        "type": "object"
      },
      "ValidationError": {
        "properties": {
          "loc": {
            "items": {
              "anyOf": [
                {
                  "type": "string"
                },
                {
                  "type": "integer"
                }
              ]
            },
            "title": "Location",
            "type": "array"
          },
          "msg": {
            "title": "Message",
            "type": "string"
          },
          "type": {
            "title": "Error Type",
            "type": "string"
          }
        },
        "required": [
          "loc",
          "msg",
          "type"
        ],
        "title": "ValidationError",
        "type": "object"
      }
    },
    "securitySchemes": {
      "JWTBearer": {
        "scheme": "bearer",
        "type": "http"
      }
    }
  },
  "info": {
    "title": "FastAPI",
    "version": "0.1.0"
  },
  "openapi": "3.0.2",
  "paths": {
    "/": {
      "get": {
        "operationId": "index__get",
        "responses": {
          "200": {
            "content": {
              "application/json": {
                "schema": {
                  "additionalProperties": {
                    "type": "string"
                  },
                  "title": "Response Index  Get",
                  "type": "object"
                }
              }
            },
            "description": "Successful Response"
          }
        },
        "summary": "Index"
      }
    },
    "/api/v1/deployments/{deployment_id}/dags/{dag_id}/structure": {
      "get": {
        "operationId": "dag_structure_api_v1_deployments__deployment_id__dags__dag_id__structure_get",
        "parameters": [
          {
            "in": "path",
            "name": "dag_id",
            "required": true,
            "schema": {
              "title": "Dag Id",
              "type": "string"
            }
          },
          {
            "in": "path",
            "name": "deployment_id",
            "required": true,
            "schema": {
              "title": "Deployment Id",
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/DagStructure"
                }
              }
            },
            "description": "Successful Response"
          },
          "422": {
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/HTTPValidationError"
                }
              }
            },
            "description": "Validation Error"
          }
        },
        "security": [
          {
            "JWTBearer": []
          }
        ],
        "summary": "Dag Structure"
      }
    }
  }
}

[ndimares]

Thanks @ashb, I see what you're trying to do. Looking into this. I think we need to make a change in our validation library. Will keep you updated!

[ndimares]

@ashb if you want to track or add any details, this is the ticket tracking the change to the validation library.

[ndimares]

@ashb One of our engineers (@TristanSpeakEasy) just pointed out to me that this should probably be using oneOf rather than anyOf. Using anyOf implies that the object can be both i.e. a Task and TaskGroup at the same time. oneOf means either or (which I think is your intent).

However I confirmed that the validation is also not working as expected if you change to oneOf, so will continue chasing that update.

[ashb]

Oh yes, fair point. I'll take that up with Pydantic!