Circular reference in OpenAPI spec is not resolved correctly

mdonkers commented 1 month ago

Hi,

Our OpenAPI spec (3.1.0) contains a circular reference that looks like:

openapi: 3.1.0
components:
  schemas:
    AnyValue:
      description: "AnyValue is used to represent any type of attribute value. AnyValue may contain a primitive value such as a string or integer or it may contain an arbitrary nested object containing arrays, key-value lists and primitives."
      type: object
      properties:
        stringValue:
          type: string
        boolValue:
          type: boolean
        intValue:
          type: string
          format: int64
        doubleValue:
          type: number
          format: double
        arrayValue:
          $ref: "#/components/schemas/ArrayValue"
        kvlistValue:
          $ref: "#/components/schemas/KeyValueList"
        bytesValue:
          type: string
          "format": byte

    ArrayValue:
      description: "ArrayValue is a list of AnyValue messages. We need ArrayValue as a message since oneof in AnyValue does not allow repeated fields."
      type: object
      properties:
        values:
          description: "Array of values. The array may be empty (contain 0 elements)."
          type: array
          items:
            $ref: "#/components/schemas/AnyValue"
      required:
        - values

We need this to correctly represent the OTLP (OpenTelemetry) wire format.

But the rdme command fails to validate or in any other way process this, with the following error:

$ rdme openapi:validate api.yml 
✖ Validating the API definition located at api.yml...

Token "ArrayValue" does not exist.

Other tools to generate both client and server endpoints correctly process the above structure.

It would be helpful if rdme could be updated to support this use case, or is there any known work-around possible? (according to the OpenAPI specs, this scenario should be valid...)

Thanks!

erunion commented 3 weeks ago

@mdonkers i'm unable to replicate this, can you possibly share the full API definition that's throwing this?

for example, this one passes validation:

openapi: 3.1.0
info:
  version: 1.0.0
  title: Simple Petstore
  description: This is a slimmed down single path version of the Petstore definition.
servers:
  - url: https://httpbin.org
paths:
  '/pet/{id}':
    parameters:
      - name: id
        in: path
        required: true
        schema:
          type: integer
    put:
      tags:
        - pet
      summary: Update a pet
      description: This operation will update a pet in the database.
      responses:
        '400':
          description: Invalid id value
      security:
        - apiKey: []
components:
  schemas:
    AnyValue:
      description: "AnyValue is used to represent any type of attribute value. AnyValue may contain a primitive value such as a string or integer or it may contain an arbitrary nested object containing arrays, key-value lists and primitives."
      type: object
      properties:
        stringValue:
          type: string
        boolValue:
          type: boolean
        intValue:
          type: string
          format: int64
        doubleValue:
          type: number
          format: double
        arrayValue:
          $ref: "#/components/schemas/ArrayValue"
        kvlistValue:
          $ref: "#/components/schemas/KeyValueList"
        bytesValue:
          type: string
          "format": byte

    KeyValueList:
      type: string

    ArrayValue:
      description: "ArrayValue is a list of AnyValue messages. We need ArrayValue as a message since oneof in AnyValue does not allow repeated fields."
      type: object
      properties:
        values:
          description: "Array of values. The array may be empty (contain 0 elements)."
          type: array
          items:
            $ref: "#/components/schemas/AnyValue"
      required:
        - value

mdonkers commented 3 weeks ago

Thanks for getting back to this. I tried to get a minimal reproducer but it was a bit more complex than I expected. There seem to be two preconditions to get this triggered:

only happens with imports from external files, not if everything is in the same file
from the objects in the circular reference, two different ones are referenced in different places

api.yaml:

openapi: 3.1.0
info:
  title: "Dash0 API: OTLP-common related typings"
  version: 1.0.0
paths:
  '/pet':
    put:
      summary: Update a pet
      description: This operation will update a pet in the database.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/SomeRequest'
      responses:
        '200':
          description: Returns a concrete timerange
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SomeResponse'
        '400':
          description: Invalid id value
components:
  schemas:
    SomeRequest:
      properties:
        filter:
          $ref: 'otlp-common.yml#/components/schemas/AttributeFilter'
    SomeResponse:
      properties:
        attributes:
          type: array
          items:
            $ref: 'otlp-common.yml#/components/schemas/KeyValue'

otlp-common.yml:

openapi: 3.1.0
info:
  title: "Dash0 API: OTLP-common related typings"
  version: 1.0.0
components:
  schemas:
    # Attribute schema definitions are coming from the OpenTelemetry OTLP wire format.
    # https://github.com/open-telemetry/opentelemetry-proto/blob/1e69bf2e789665a56eed3a3e9cf4fd51ff784ab6/opentelemetry/proto/common/v1/common.proto#LL28C1-L28C1
    # and the attribute spec
    # https://opentelemetry.io/docs/specs/otel/common/#attribute
    #

    AnyValue:
      description: "AnyValue is used to represent any type of attribute value. AnyValue may contain a primitive value such as a string or integer or it may contain an arbitrary nested object containing arrays, key-value lists and primitives."
      type: object
      properties:
        stringValue:
          type: string
        boolValue:
          type: boolean
        intValue:
          type: string
          format: int64
        doubleValue:
          type: number
          format: double
        arrayValue:
          $ref: "#/components/schemas/ArrayValue"
        kvlistValue:
          $ref: "#/components/schemas/KeyValueList"
        bytesValue:
          type: string
          "format": byte

    ArrayValue:
      description: "ArrayValue is a list of AnyValue messages. We need ArrayValue as a message since oneof in AnyValue does not allow repeated fields."
      type: object
      properties:
        values:
          description: "Array of values. The array may be empty (contain 0 elements)."
          type: array
          items:
            $ref: "#/components/schemas/AnyValue"
      required:
        - values

    KeyValue:
      description: "KeyValue is a key-value pair that is used to store Span attributes, Link attributes, etc."
      type: object
      properties:
        key:
          type: string
        value:
          $ref: "#/components/schemas/AnyValue"
      required:
        - key
        - value

    KeyValueList:
      description: "KeyValueList is a list of KeyValue messages. We need KeyValueList as a message since `oneof` in AnyValue does not allow repeated fields. Everywhere else where we need a list of KeyValue messages (e.g. in Span) we use `repeated KeyValue` directly to avoid unnecessary extra wrapping (which slows down the protocol). The 2 approaches are semantically equivalent."
      type: object
      properties:
        values:
          description: "A collection of key/value pairs of key-value pairs. The list may be empty (may contain 0 elements). The keys MUST be unique (it is not allowed to have more than one value with the same key)."
          type: array
          items:
            $ref: "#/components/schemas/KeyValue"
      required:
        - values

    AttributeFilterKey:
      type: string
      description: The attribute key to be filtered.

    AttributeFilterAnyValue:
      description: "AttributeFilterAnyValue may contain any AnyValue value."
      $ref: 'otlp-common.yml#/components/schemas/AnyValue'

    AttributeFilter:
      type: object
      properties:
        key:
          $ref: '#/components/schemas/AttributeFilterKey'
        value:
          $ref: '#/components/schemas/AttributeFilterAnyValue'

Then running rdme openapi:validate api.yml will give the error:

✖ Validating the API definition located at api.yml...

Token "KeyValue" does not exist.

erunion commented 3 weeks ago

Ah interesting, it may be that the reducer isn't running bundling in the right directory and is unable to properly load otlp-common.yml.

mdonkers commented 3 weeks ago

Diving into this further, and seeing it only happens with the imports, I'm getting to think the other issue I posted might very well be related: https://github.com/readmeio/rdme/issues/1053 Just FYI.

readmeio / rdme

Circular reference in OpenAPI spec is not resolved correctly #1052