schemathesis / schemathesis

Supercharge your API testing, catch bugs, and ensure compliance
https://schemathesis.readthedocs.io
MIT License
2.25k stars 161 forks source link

[BUG] OpenAPI 3.0: shared example generating invalid case #2278

Closed ravy closed 3 months ago

ravy commented 3 months ago

Checklist

Describe the bug

When using shared examples in OpenAPI 3.0. and using option --hypothesis-phases=expicit Then there is an invalid test case being generated. By default the DataGenerationMethod of Hypothesis should be 'positive' so was surprised to see this invalid case.

To Reproduce

  1. save given api schema as testopenapi30.yaml
  2. Run this command st run -v --hypothesis-phases=explicit --validate-schema=true -c all --hypothesis-verbosity=debug --base-url=http://localhost:8083 --dry-run testopenapi30.yaml
  3. See error, second case is an invalid case. Case(headers={'Authorization': 'Bearer '}, body={'title': 'Running', 'description': 'run a marathon'}) Case(headers={'Authorization': 'Bearer '}, body={'value': {'title': 'Running', 'description': 'run a marathon'}})

Please include a minimal API schema causing this issue:

openapi: '3.0.3'
info:
  version: 0.1.0
  title: Item List API
  license:
    name: 'Test'
servers:
  -
    url: http://localhost:8083
# apply security to all operations
security:
  -
    bearerAuth: []

paths:
  /items:
    post:
      summary: Add a new item to the list
      operationId: addItem
      tags:
        - items
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Item'
            examples:
              fitnessItem:
                $ref: "#/components/examples/fitnessItem"
      responses:
        '201':
          description: Item added successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Item'
        '4XX':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '5XX':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          $ref: '#/components/responses/UnauthorizedError'
components:
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
  responses:
    UnauthorizedError:
      description: Access token is missing or invalid
  schemas:
    Items:
      type: array
      items:
        $ref: '#/components/schemas/Item'
    Error:
      type: object
      required:
        - message
      properties:
        message:
          type: string
        data:
          type: object
    Item:
      type: object
      required:
        - title
        - description
      properties:
        id:
          type: string
          description: ID of the list item.
          format: uuid
        title:
          type: string
          description: Title of the item.
        description:
          type: string
          description: More detailed description of the item.
        year:
          type: string
          description: Target year
          pattern: '^\d{4}'
  examples:
    fitnessItem:
      value:
        title: Running
        description: run a marathon

Expected behavior

There should be only one valid explicit example generated.

Environment

- OS: Linux
- Python version: 3.9.18
- Schemathesis version: 3.29.2 (hypothesis-6.103.1, hypothesis_jsonschema-0.23.1, jsonschema-4.22.0)
- Spec version: Open API 2.0

Additional context

this is not such a serious issue as the second case can be considered a negative test case that does not conform to schema. but it is unintended while testing only examples. I also tried using OpenAPI 3.1 with the --experimental=openapi-3.1 flag set, but the same issue exists.

Stranger6667 commented 3 months ago

Hi! I think it is a duplicate of #2238. With the latest commit I can't reproduce the issue - Schemathesis generates exactly 1 example:

Case(headers={'Authorization': 'Bearer '}, body={'title': 'Running', 'description': 'run a marathon'})