fix: [openapi31] Fixes #146 when anyOf is used in schemas.

cedricbonhomme commented 10 months ago

Change to address the issue #146. Description of the problem is in the issue.

Tested on my side.

Let me know if you will merge it. I need to deploy a documentation with sphinx-contrib-openapi on ReadTheDocs.org.

Thank you very much.

stephenfin commented 9 months ago

Could we get a test for this? You can see examples of this in e.g. commit 803da3f45745c5cc8fba8d570fbbf3dba11c948a.

cedricbonhomme commented 9 months ago

Would it be OK if I push a simplified version of my schema (as yaml) ? This one:

---
openapi: "3.1.0"
info:
  title: "Reproducer for issue #147"
components:
  schemas:
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          title: Detail
          type: array
      title: HTTPValidationError
      type: object
    ItemBase:
      properties:
        scan_data:
          $ref: '#/components/schemas/ScanData'
      required:
      - scan_data
      title: ItemBase
      type: object
    Meta:
      properties:
        ts:
          title: Ts
          type: integer
        type:
          title: Type
          type: string
        uuid:
          title: Uuid
          type: string
      required:
      - uuid
      - ts
      - type
      title: Meta
      type: object
    Payload:
      properties:
        row:
          title: Row
          type: string
      required:
      - row
      title: Payload
      type: object
    ScanData:
      properties:
        format:
          title: Format
          type: string
        meta:
          $ref: '#/components/schemas/Meta'
        payload:
          $ref: '#/components/schemas/Payload'
        version:
          title: Version
          type: string
      required:
      - version
      - format
      - meta
      - payload
      title: ScanData
      type: object
    ScanDataCreate:
      properties:
        format:
          title: Format
          type: string
        meta:
          $ref: '#/components/schemas/Meta'
        payload:
          $ref: '#/components/schemas/Payload'
        version:
          title: Version
          type: string
      required:
      - version
      - format
      - meta
      - payload
      title: ScanDataCreate
      type: object
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
            - type: string
            - type: integer
          title: Location
          type: array
        msg:
          title: Message
          type: string
        type:
          title: Error Type
          type: string
      required:
      - loc
      - msg
      - type
      title: ValidationError
      type: object
  securitySchemes:
    HTTPBasic:
      scheme: basic
      type: http
paths:
  /TimeStampTokens/:
    get:
      operationId: read_tsts_TimeStampTokens__get
      parameters:
      - in: query
        name: skip
        required: false
        schema:
          default: 0
          title: Skip
          type: integer
      - in: query
        name: limit
        required: false
        schema:
          default: 100
          title: Limit
          type: integer
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Read Tsts
    post:
      description: Insert a TimeStampToken and publish it through the WebSocket.
      operationId: create_tst_TimeStampTokens__post
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
      security:
      - HTTPBasic: []
      summary: Create Tst
  /TimeStampTokens/check/{scan_uuid}:
    get:
      description: Performs an offline check of a TimeStampToken.
      operationId: check_tst_TimeStampTokens_check__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Check Tst
  /TimeStampTokens/get_timestamp/{scan_uuid}:
    get:
      description: Get the timestamp of a check based on the corresponding TimeStampToken
        object.
      operationId: get_timestamp_TimeStampTokens_get_timestamp__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema: {}
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Get Timestamp
  /TimeStampTokens/token/{scan_uuid}:
    get:
      description: Returns directly the TST as a raw binary stream corresponding to
        the scan UUID given as parameter.
      operationId: get_tst_token_TimeStampTokens_token__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema:
                format: binary
                title: Response Get Tst Token Timestamptokens Token  Scan Uuid  Get
                type: string
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Get Tst Token
  /TimeStampTokens/{scan_uuid}:
    get:
      description: Returns a TimeStampToken object corresponding to the scan UUID
        given as parameter.
      operationId: get_tst_TimeStampTokens__scan_uuid__get
      parameters:
      - in: path
        name: scan_uuid
        required: true
        schema:
          title: Scan Uuid
      responses:
        '200':
          content:
            application/json:
              schema:
                format: binary
                title: Response Get Tst Timestamptokens  Scan Uuid  Get
                type: string
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Get Tst
  /items/:
    get:
      operationId: read_items_items__get
      parameters:
      - in: query
        name: skip
        required: false
        schema:
          default: 0
          title: Skip
          type: integer
      - in: query
        name: limit
        required: false
        schema:
          default: 100
          title: Limit
          type: integer
      - in: query
        name: q
        required: false
        schema:
          default: ''
          title: Q
          type: string
      responses:
        '200':
          content:
            application/json:
              schema:
                items:
                  $ref: '#/components/schemas/ItemBase'
                title: Response Read Items Items  Get
                type: array
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Read Items
    post:
      description: Insert a new item and publish it through the WebSocket.
      operationId: create_item_items__post
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ScanDataCreate'
        required: true
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ItemBase'
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      security:
      - HTTPBasic: []
      summary: Create Item
  /items/{item_id}:
    get:
      operationId: read_item_items__item_id__get
      parameters:
      - in: path
        name: item_id
        required: true
        schema:
          title: Item Id
          type: integer
      responses:
        '200':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ItemBase'
          description: Successful Response
        '422':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
          description: Validation Error
      summary: Read Item

stephenfin commented 9 months ago

Would it be OK if I push a simplified version of my schema (as yaml) ? This one:
(snip)

Yup, no issues. You can obviously simplify that a lot since the anyOf bit is what we're testing.

cedricbonhomme commented 9 months ago

I've just pushed a version more simplified ! let me know if it is OK!

stephenfin commented 9 months ago

I've just pushed a version more simplified ! let me know if it is OK!

I don't think it is, unfortunately.

$ tox -e py310 -q
============================================================================== test session starts ===============================================================================
...
============================================================================== 495 passed in 7.96s ===============================================================================
  py310: OK (10.21 seconds)
  congratulations :) (10.25 seconds)

$ git revert HEAD~

$ tox -e py310 -q
============================================================================== test session starts ===============================================================================
...
============================================================================== 495 passed in 8.23s ===============================================================================
  py310: OK (10.55=setup[2.02]+cmd[8.53] seconds)
  congratulations :) (10.60 seconds)

I would expect the new test to fail without the change. Could you rework that spec so this happens? :pray:

cedricbonhomme commented 9 months ago

Of course, I'll try to check locally this time.

cedricbonhomme commented 9 months ago

The complete yaml file which generates an error when generating the documentation do not generates an error with the tests.

(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ make html 
Running Sphinx v7.2.6
checking bibtex cache... out of date
parsing bibtex file /home/cedric/git/SCANDAL/scandale/docs/refs.bib... parsed 8 entries
The default value for `navigation_with_keys` will change to `False` in the next release. If you wish to preserve the old behavior for your site, set `navigation_with_keys=True` in the `html_theme_options` dict in your `conf.py` file. Be aware that `navigation_with_keys = True` has negative accessibility implications: https://github.com/pydata/pydata-sphinx-theme/issues/1492
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 6 source files that are out of date
updating environment: [new config] 6 added, 0 changed, 0 removed
reading sources... [ 50%] formats
Exception occurred:
  File "/home/cedric/.cache/pypoetry/virtualenvs/scandale-YR99gNwR-py3.10/lib/python3.10/site-packages/sphinxcontrib/openapi/openapi31.py", line 305, in _get_type_from_schema
    for t in schema["anyOf"]:
KeyError: 'anyOf'
The full traceback has been saved in /tmp/sphinx-err-1e8d5c9f.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:20: html] Error 2
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ 
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ 
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ 
(scandale-py3.10) cedric@debian:~/git/SCANDAL/scandale/docs$ sha1sum _static/openapi.yaml 
bbe7a7fab39d11d8d1a328318a1c10f0091ce8ca  _static/openapi.yaml

cedric@debian:~/git/openapi$ sha1sum tests/testspecs/v3.1/issue-147.yml 
bbe7a7fab39d11d8d1a328318a1c10f0091ce8ca  tests/testspecs/v3.1/issue-147.yml
cedric@debian:~/git/openapi$ tox -e py312 -q
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0
cachedir: .tox/py312/.pytest_cache
rootdir: /home/cedric/git/openapi
configfile: tox.ini
collected 509 items                                                                                                                                                                                      

tests/test_openapi.py ..................................                                                                                                                                           [  6%]
tests/test_schema_utils.py .........                                                                                                                                                               [  8%]
tests/test_spec_examples.py ..................................................................                                                                                                     [ 21%]
tests/lib2to3/test_convert.py ............                                                                                                                                                         [ 24%]
tests/lib2to3/test_convert_operation.py ......                                                                                                                                                     [ 25%]
tests/lib2to3/test_convert_parameter.py ................                                                                                                                                           [ 28%]
tests/lib2to3/test_convert_parameters.py ...                                                                                                                                                       [ 29%]
tests/lib2to3/test_convert_path.py ...........                                                                                                                                                     [ 31%]
tests/lib2to3/test_convert_paths.py ....                                                                                                                                                           [ 32%]
tests/lib2to3/test_convert_request_body.py .....                                                                                                                                                   [ 33%]
tests/lib2to3/test_convert_request_body_formdata.py ............                                                                                                                                   [ 35%]
tests/lib2to3/test_convert_response.py .............                                                                                                                                               [ 38%]
tests/lib2to3/test_convert_responses.py ....                                                                                                                                                       [ 39%]
tests/renderers/httpdomain/test_render.py .............                                                                                                                                            [ 41%]
tests/renderers/httpdomain/test_render_json_schema_description.py ..................................................................................................                               [ 61%]
tests/renderers/httpdomain/test_render_operation.py .............                                                                                                                                  [ 64%]
tests/renderers/httpdomain/test_render_parameter.py ..........................................                                                                                                     [ 72%]
tests/renderers/httpdomain/test_render_parameters.py .............                                                                                                                                 [ 75%]
tests/renderers/httpdomain/test_render_paths.py .........                                                                                                                                          [ 77%]
tests/renderers/httpdomain/test_render_request_body.py ....                                                                                                                                        [ 78%]
tests/renderers/httpdomain/test_render_request_body_example.py .........................                                                                                                           [ 83%]
tests/renderers/httpdomain/test_render_response.py ....................................                                                                                                            [ 90%]
tests/renderers/httpdomain/test_render_response_example.py ..............................                                                                                                          [ 96%]
tests/renderers/httpdomain/test_render_responses.py ........                                                                                                                                       [ 97%]
tests/renderers/httpdomain/test_render_restructuredtext_markup.py ..........                                                                                                                       [100%]

============================================================================================ warnings summary ============================================================================================
sphinxcontrib/openapi/utils.py:27
  /home/cedric/git/openapi/sphinxcontrib/openapi/utils.py:27: DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
    class OpenApiRefResolver(jsonschema.RefResolver):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 496 passed, 1 warning in 9.05s =====================================================================================
  py312: OK (13.55 seconds)
  congratulations :) (13.59 seconds)

So I guess, that's why.

stephenfin commented 9 months ago

Interesting. Must be Sphinx-specific logic at play so. I'll try to take a look this weekend.

cedricbonhomme commented 9 months ago

Thank you. Let me know if I can help. I can go deeper.

As you can see the error is raised at line 305, file openapi/openapi31.py.

stephenfin commented 9 months ago

Okay, this is happening because the tests are using the new renderer - which is not enabled by default - instead of the old renderer. I guess we need to switch that over at some point. Until then, this is fine as-is. Let's just see if we can get some tests of the old renderer test going first.

cbecker commented 5 months ago

I'm also hitting this error and crashes during generation. Any chances this PR will get merged soon?

cedricbonhomme commented 4 months ago

I'm also hitting this error and crashes during generation. Any chances this PR will get merged soon?

This would be great yes ! ;-)

pankajrlal commented 4 months ago

@stephenfin When can we see this merge? It is affecting our builds

stephenfin commented 4 months ago

Let's just see if we can get some tests of the old renderer test going first.

:point_up: We need some tests running against the old renderer first otherwise the tests are useless. Once that's addressed I'm happy to merge this.

sphinx-contrib / openapi

fix: [openapi31] Fixes #146 when anyOf is used in schemas. #147