fix: [openapi31] Fixes #146 when anyOf is used in schemas.

cedricbonhomme commented 10 months ago

Change to address the issue #146. Description of the problem is in the issue.

Tested on my side.

Let me know if you will merge it. I need to deploy a documentation with sphinx-contrib-openapi on ReadTheDocs.org.

Thank you very much.

stephenfin commented 10 months ago

Could we get a test for this? You can see examples of this in e.g. commit 803da3f45745c5cc8fba8d570fbbf3dba11c948a.

cedricbonhomme commented 10 months ago

Would it be OK if I push a simplified version of my schema (as yaml) ? This one:

---
openapi: "3.1.0"
info:
  title: "Reproducer for issue #147"
components:
  schemas:
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          title: Detail
          type: array
      title: HTTPValidationError
      type: object
    ItemBase:
      properties:
        scan_data:
          $ref: '#/components/schemas/ScanData'
      required:
      - scan_data
      title: ItemBase
      type: object
    Meta:
      properties:
        ts:
          title: Ts
          type: integer
        type:
          title: Type
          type: string
        uuid:
          title: Uuid
          type: string
      required:
      - uuid
      - ts
      - type
      title: Meta
      type: object
    Payload:
      properties:
        row:
          title: Row
          type: string
      required:
      - row
      title: Payload
      type: object
    ScanData:
      properties:
        format:
          title: Format
          type: string
        meta:
          $ref: '#/components/schemas/Meta'
        payload:
          $ref: '#/components/schemas/Payload'
        version:
          title: Version
          type: string
      required:
      - version
      - format
      - meta
      - payload
      title: ScanData
      type: object
    ScanDataCreate:
      properties:
        format:
          title: Format
          type: string
        meta:
          $ref: '#/components/schemas/Meta'
        payload:
          $ref: '#/components/schemas/Payload'
        version:
          title: Version
          type: string
      required:
      - version
      - format
      - meta
      - payload
      title: ScanDataCreate
      type: object
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
            - type: string
            - type: integer
          title: Location
          type: array
        msg:
          title: Message
          type: string
        type:
          title: Error Type
          type: string
      required:
      - loc
      - msg
      - type
      title: ValidationError
      type: object
  securitySchemes:
    HTTPBasic:
      scheme: basic
      type: http
paths:
  /TimeStampTokens/:
    get:
      operationId: read_tsts_TimeStampTokens__get
      parameters:
      - in: query
        name: skip
        required: false
        schema:
          default: 0
          title: Skip
          type: integer
      - in: query
        name: limit
        required: false
        schema:
          default: 100
          title: Limit
          type: integer
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Read Tsts
    post:
      description: Insert a TimeStampToken and publish it through the WebSocket.
      operationId: create_tst_TimeStampTokens__post
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
      security:
      - HTTPBasic: []
      summary: Create Tst
  /TimeStampTokens/check/{scan_uuid}:
    get:
      description: Performs an offline check of a TimeStampToken.
      operationId: check_tst_TimeStampTokens_check__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Check Tst
  /TimeStampTokens/get_timestamp/{scan_uuid}:
    get:
      description: Get the timestamp of a check based on the corresponding TimeStampToken
        object.
      operationId: get_timestamp_TimeStampTokens_get_timestamp__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Get Timestamp
  /TimeStampTokens/token/{scan_uuid}:
    get:
      description: Returns directly the TST as a raw binary stream corresponding to
        the scan UUID given as parameter.
      operationId: get_tst_token_TimeStampTokens_token__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema:
                format: binary
                title: Response Get Tst Token Timestamptokens Token  Scan Uuid  Get
                type: string
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Get Tst Token
  /TimeStampTokens/{scan_uuid}:
    get:
      description: Returns a TimeStampToken object corresponding to the scan UUID
        given as parameter.
      operationId: get_tst_TimeStampTokens__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema:
                format: binary
                title: Response Get Tst Timestamptokens  Scan Uuid  Get
                type: string
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Get Tst
  /items/:
    get:
      operationId: read_items_items__get
      parameters:
      - in: query
        name: skip
        required: false
        schema:
          default: 0
          title: Skip
          type: integer
      - in: query
        name: limit
        required: false
        schema:
          default: 100
          title: Limit
          type: integer
      - in: query
        name: q
        required: false
        schema:
          default: ''
          title: Q
          type: string
      responses:
        '200':
          content:
            application/json:
              schema:
                items:
                  $ref: '#/components/schemas/ItemBase'
                title: Response Read Items Items  Get
                type: array
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Read Items
    post:
      description: Insert a new item and publish it through the WebSocket.
      operationId: create_item_items__post
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ScanDataCreate'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ItemBase'
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      security:
      - HTTPBasic: []
      summary: Create Item
  /items/{item_id}:
    get:
      operationId: read_item_items__item_id__get
      parameters:
      - in: path
        name: item_id
        required: true
        schema:
          title: Item Id
          type: integer
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ItemBase'
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Read Item

stephenfin commented 10 months ago

Would it be OK if I push a simplified version of my schema (as yaml) ? This one:
(snip)

Yup, no issues. You can obviously simplify that a lot since the anyOf bit is what we're testing.

cedricbonhomme commented 10 months ago

I've just pushed a version more simplified ! let me know if it is OK!

stephenfin commented 10 months ago

I've just pushed a version more simplified ! let me know if it is OK!

I don't think it is, unfortunately.

$ tox -e py310 -q
============================================================================== test session starts ===============================================================================
...
============================================================================== 495 passed in 7.96s ===============================================================================
  py310: OK (10.21 seconds)
  congratulations :) (10.25 seconds)

$ git revert HEAD~

$ tox -e py310 -q
============================================================================== test session starts ===============================================================================
...
============================================================================== 495 passed in 8.23s ===============================================================================
  py310: OK (10.55=setup[2.02]+cmd[8.53] seconds)
  congratulations :) (10.60 seconds)

I would expect the new test to fail without the change. Could you rework that spec so this happens? :pray:

cedricbonhomme commented 10 months ago

Of course, I'll try to check locally this time.

cedricbonhomme commented 10 months ago

The complete yaml file which generates an error when generating the documentation do not generates an error with the tests.

(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ make html 
Running Sphinx v7.2.6
checking bibtex cache... out of date
parsing bibtex file /home/cedric/git/SCANDAL/scandale/docs/refs.bib... parsed 8 entries
The default value for `navigation_with_keys` will change to `False` in the next release. If you wish to preserve the old behavior for your site, set `navigation_with_keys=True` in the `html_theme_options` dict in your `conf.py` file. Be aware that `navigation_with_keys = True` has negative accessibility implications: https://github.com/pydata/pydata-sphinx-theme/issues/1492
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 6 source files that are out of date
updating environment: [new config] 6 added, 0 changed, 0 removed
reading sources... [ 50%] formats
Exception occurred:
  File "/home/cedric/.cache/pypoetry/virtualenvs/scandale-YR99gNwR-py3.10/lib/python3.10/site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema
    for t in schema["anyOf"]:
KeyError: 'anyOf'
The full traceback has been saved in /tmp/sphinx-err-1e8d5c9f.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:20: html] Error 2
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ 
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ 
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ 
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ sha1sum _static/openapi.yaml 
bbe7a7fab39d11d8d1a328318a1c10f0091ce8ca  _static/openapi.yaml

cedric@debian:~/git/openapi$ sha1sum tests/testspecs/v3.1/issue-147.yml 
bbe7a7fab39d11d8d1a328318a1c10f0091ce8ca  tests/testspecs/v3.1/issue-147.yml
cedric@debian:~/git/openapi$ tox -e py312 -q
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0
cachedir: .tox/py312/.pytest_cache
rootdir: /home/cedric/git/openapi
configfile: tox.ini
collected 509 items                                                                                                                                                                                      

tests/test_openapi.py ..................................                                                                                                                                           [  6%]
tests/test_schema_utils.py .........                                                                                                                                                               [  8%]
tests/test_spec_examples.py ..................................................................                                                                                                     [ 21%]
tests/lib2to3/test_convert.py ............                                                                                                                                                         [ 24%]
tests/lib2to3/test_convert_operation.py ......                                                                                                                                                     [ 25%]
tests/lib2to3/test_convert_parameter.py ................                                                                                                                                           [ 28%]
tests/lib2to3/test_convert_parameters.py ...                                                                                                                                                       [ 29%]
tests/lib2to3/test_convert_path.py ...........                                                                                                                                                     [ 31%]
tests/lib2to3/test_convert_paths.py ....                                                                                                                                                           [ 32%]
tests/lib2to3/test_convert_request_body.py .....                                                                                                                                                   [ 33%]
tests/lib2to3/test_convert_request_body_formdata.py ............                                                                                                                                   [ 35%]
tests/lib2to3/test_convert_response.py .............                                                                                                                                               [ 38%]
tests/lib2to3/test_convert_responses.py ....                                                                                                                                                       [ 39%]
tests/renderers/httpdomain/test_render.py .............                                                                                                                                            [ 41%]
tests/renderers/httpdomain/test_render_json_schema_description.py ..................................................................................................                               [ 61%]
tests/renderers/httpdomain/test_render_operation.py .............                                                                                                                                  [ 64%]
tests/renderers/httpdomain/test_render_parameter.py ..........................................                                                                                                     [ 72%]
tests/renderers/httpdomain/test_render_parameters.py .............                                                                                                                                 [ 75%]
tests/renderers/httpdomain/test_render_paths.py .........                                                                                                                                          [ 77%]
tests/renderers/httpdomain/test_render_request_body.py ....                                                                                                                                        [ 78%]
tests/renderers/httpdomain/test_render_request_body_example.py .........................                                                                                                           [ 83%]
tests/renderers/httpdomain/test_render_response.py ....................................                                                                                                            [ 90%]
tests/renderers/httpdomain/test_render_response_example.py ..............................                                                                                                          [ 96%]
tests/renderers/httpdomain/test_render_responses.py ........                                                                                                                                       [ 97%]
tests/renderers/httpdomain/test_render_restructuredtext_markup.py ..........                                                                                                                       [100%]

============================================================================================ warnings summary ============================================================================================
sphinxcontrib/openapi/utils.py:27
  /home/cedric/git/openapi/sphinxcontrib/openapi/utils.py:27: DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
    class OpenApiRefResolver(jsonschema.RefResolver):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 496 passed, 1 warning in 9.05s =====================================================================================
  py312: OK (13.55 seconds)
  congratulations :) (13.59 seconds)

So I guess, that's why.

stephenfin commented 10 months ago

Interesting. Must be Sphinx-specific logic at play so. I'll try to take a look this weekend.

cedricbonhomme commented 10 months ago

Thank you. Let me know if I can help. I can go deeper.

As you can see the error is raised at line 305, file openapi/openapi31.py.

stephenfin commented 10 months ago

Okay, this is happening because the tests are using the new renderer - which is not enabled by default - instead of the old renderer. I guess we need to switch that over at some point. Until then, this is fine as-is. Let's just see if we can get some tests of the old renderer test going first.

cbecker commented 5 months ago

I'm also hitting this error and crashes during generation. Any chances this PR will get merged soon?

cedricbonhomme commented 5 months ago

I'm also hitting this error and crashes during generation. Any chances this PR will get merged soon?

This would be great yes ! ;-)

pankajrlal commented 4 months ago

@stephenfin When can we see this merge? It is affecting our builds

stephenfin commented 4 months ago

Let's just see if we can get some tests of the old renderer test going first.

:point_up: We need some tests running against the old renderer first otherwise the tests are useless. Once that's addressed I'm happy to merge this.

sphinx-contrib / openapi

fix: [openapi31] Fixes #146 when anyOf is used in schemas. #147