search

Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.17k stars 74 forks source link

Error in payload generation #66

Closed a-waider closed 1 year ago

a-waider commented 1 year ago

Hi there,

I noticed a bug during the payload generation. Let's assume following openAPI.yaml document: 

info:
  description: ""
  title: ""
  version: ""
openapi: 3.0.1
paths:
  /api/some-endpoint:
    post:
      requestBody:
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/someRequestBody"
        required: true
      responses:
        "200":
          content:
            application/json:
              schema:
                type: object
          description: OK
components:
  schemas:
    someRequestBody:
      properties:
        someRequestBodyKey1:
          $ref: "#/components/schemas/someObject"
        someRequestBodyKey2:
          format: date-time
          type: string
        someRequestBodyKey3:
          items:
            format: int32
            type: integer
          type: array
        someRequestBodyKey4:
          format: date-time
          type: string
      type: object
    someObject:
      properties:
        someObjectKey1:
          anyOf:
            - $ref: "#/components/schemas/someSubObject1"
            - $ref: "#/components/schemas/someSubObject2"
          type: object
        someObjectKey2:
          format: int32
          type: integer
        someObjectKey3:
          type: string
        someObjectKey4:
          type: string
      required:
        - someObjectKey1
        - someObjectKey2
        - someObjectKey3
        - someObjectKey4
      type: object
    someSubObject1:
      properties:
        someSubObjectKey1:
          items:
            format: double
            type: number
          maxItems: 2
          minItems: 2
          type: array
        someSubObjectKey2:
          format: double
          type: number
        someSubObjectKey3:
          type: string
      required:
        - someSubObjectKey1
        - someSubObjectKey2
        - someSubObjectKey3
      type: object
    someSubObject2:
      properties:
        someSubObjectKey1:
          items:
            format: double
            type: number
          type: array
        someSubObjectKey2:
          type: string
      required:
        - someSubObjectKey1
        - someSubObjectKey2
      type: object

After executing CATS with cats --contract openAPI.yaml --server http://localhost:8080 --fuzzers CheckSecurityHeaders there are 5 tests. Some of the test have invalid request payloads for someRequestBodyKey1, e.g. 

{
  "someRequestBodyKey4": "2023-04-21T15:10:26.521072Z",
  "someRequestBodyKey3": [
    8,
    8
  ],
  "someRequestBodyKey2": "2023-04-21T15:10:26.520904Z",
  "someRequestBodyKey1": 7
}

As per openAPI.yml definition there should be an object type and not a number or string. That wouldn't be a problem if the expected response code would be HTTP 400 but CATS expects HTTP 200.

The same behaviour happens to multiple fuzzers so I'm guessing it is not a problem with any specific fuzzer.

Any idea what could be the problem?

en-milie commented 1 year ago

Hi @a-waider. This seems to be an issue. I'll take a look and get back with a fix.

en-milie commented 1 year ago

This should be now fixed in: https://github.com/Endava/cats/releases/tag/cats-8.6.0

a-waider commented 1 year ago

I don't see the specific issue anymore in my reports, so I would assume it's fixed. Thanks :)

Therefore I noticed another issue which seems to be related to this one. Consider this OpenAPI.yaml file:

info:
  description: ""
  title: ""
  version: ""
openapi: 3.0.1
paths:
  /api/some-endpoint:
    post:
      requestBody:
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/someRequestBody"
        required: true
      responses:
        "200":
          content:
            application/json:
              schema:
                type: object
          description: OK
components:
  schemas:
    someRequestBody:
      properties:
        someRequestBodyKey1:
          $ref: "#/components/schemas/someObject"
        someRequestBodyKey2:
          format: date-time
          type: string
        someRequestBodyKey3:
          items:
            format: int32
            type: integer
          type: array
        someRequestBodyKey4:
          format: date-time
          type: string
      type: object
    someObject:
      properties:
        someObjectKey1:
          anyOf:
            - $ref: "#/components/schemas/someSubObject1"
            - $ref: "#/components/schemas/someSubObject2"
          type: object
        someObjectKey2:
          format: int32
          type: integer
        someObjectKey3:
          type: string
        someObjectKey4:
          type: string
      required:
        - someObjectKey1
        - someObjectKey2
        - someObjectKey3
        - someObjectKey4
      type: object
    someSubObject1:
      properties:
        someSubObjectKey1:
          type: array
          minItems: 1
          items:
            type: array
            minItems: 4
            items:
              type: array
              minItems: 2
              maxItems: 2
              items:
                type: number
        someSubObjectKey2:
          format: double
          type: number
        someSubObjectKey3:
          type: string
      required:
        - someSubObjectKey1
        - someSubObjectKey2
        - someSubObjectKey3
      type: object
    someSubObject2:
      properties:
        someSubObjectKey1:
          items:
            format: double
            type: number
          type: array
        someSubObjectKey2:
          type: string
      required:
        - someSubObjectKey1
        - someSubObjectKey2
      type: object

As a background information someSubObjectKey1 is some kind of Polygon as in geojson. so the lowest array must always contain exactly 2 items and the second lowest array must contain at least 4 items. When executed as in the first post (cats --contract openAPI.yaml --server http://localhost:8080 --fuzzers CheckSecurityHeaders) both created test cases have an invalid request body: Test 1 (missmatch from someSubObjectKey1[0].length=2 != minItems=4):

{
  "someRequestBodyKey4": "2023-04-27T07:56:54.504217Z",
  "someRequestBodyKey3": [
    6,
    6
  ],
  "someRequestBodyKey2": "2023-04-27T07:56:54.50408Z",
  "someRequestBodyKey1": {
    "someObjectKey3": "V79oieNYgVHNsIO",
    "someObjectKey2": 2,
    "someObjectKey4": "J05awFtvOyc5arhJ",
    "someObjectKey1": {
      "someSubObjectKey3": "AcLmp8TijB6utBMHJEaIa",
      "someSubObjectKey2": 3.1144942833214095,
      "someSubObjectKey1": [
        [
          [
            5.45746504795749,
            5.45746504795749
          ],
          [
            5.45746504795749,
            5.45746504795749
          ]
        ],
        [
          [
            5.45746504795749,
            5.45746504795749
          ],
          [
            5.45746504795749,
            5.45746504795749
          ]
        ]
      ]
    }
  }
}

Test 2 (invalid schema for someSubObjectKey1. Should be an array of array of array of numbers):

{
  "someRequestBodyKey4": "2023-04-27T07:56:54.504217Z",
  "someRequestBodyKey3": [
    6,
    6
  ],
  "someRequestBodyKey2": "2023-04-27T07:56:54.50408Z",
  "someRequestBodyKey1": {
    "someObjectKey3": "V79oieNYgVHNsIO",
    "someObjectKey2": 2,
    "someObjectKey4": "J05awFtvOyc5arhJ",
    "someObjectKey1": {
      "someSubObjectKey2": "BjZiyBPOY2gUiSKHTs1",
      "someSubObjectKey1": [
        1.09954524994883,
        1.09954524994883
      ]
    }
  }
}
en-milie commented 1 year ago

This should be fixed in this commit. The generated payload now looks like this:

{
  "someRequestBodyKey4": "2023-04-27T17:31:20.409499Z",
  "someRequestBodyKey3": [
    4,
    6
  ],
  "someRequestBodyKey2": "2023-04-27T17:31:20.407417Z",
  "someRequestBodyKey1": {
    "someObjectKey3": "MwOUAqb5Nt3tzANj",
    "someObjectKey2": 9,
    "someObjectKey4": "gj77qqkdoxKqT6YZaC",
    "someObjectKey1": {
      "someSubObjectKey3": "IUJ7NNRtIwYfLhLonKJ",
      "someSubObjectKey2": 9.884209461111075,
      "someSubObjectKey1": [
        [
          [
            6.8105032488649275,
            4.397817468372174
          ],
          [
            8.585113370015788,
            7.704676630012816
          ],
          [
            4.3858959193459395,
            4.983685438324637
          ],
          [
            3.0659290008332905,
            9.536904482968643
          ]
        ]
      ]
    }
  }
}

But I struggle though to identify why the second generated payload is invalid (Test 2 above). The OpenAPI schemas says that someObjectKey can be:

        someObjectKey1:
          anyOf:
            - $ref: "#/components/schemas/someSubObject1"
            - $ref: "#/components/schemas/someSubObject2"

someSubObject is now properly generated.

someSubObject2 has a string property and a simple array. So the generated payload is:

{
  "someRequestBodyKey4": "2023-04-27T17:31:20.409499Z",
  "someRequestBodyKey3": [
    4,
    6
  ],
  "someRequestBodyKey2": "2023-04-27T17:31:20.407417Z",
  "someRequestBodyKey1": {
    "someObjectKey3": "MwOUAqb5Nt3tzANj",
    "someObjectKey2": 9,
    "someObjectKey4": "gj77qqkdoxKqT6YZaC",
    "someObjectKey1": {
      "someSubObjectKey2": "xUxomfVcCcE5BZ3HH24",
      "someSubObjectKey1": [
        4.213428829125082,
        0.6776255436807921
      ]
    }
  }
}

Am I missing something?

a-waider commented 1 year ago

Yeah, you are right Test 2 is correct on the given example. I completely forgot that someSubObject2 is also possible. Thanks for your quick fix. 🥳 If I find anything more I will come back to you.

en-milie commented 1 year ago

Great 👍 Thank you for reporting this. It helps seeing as many use cases as possible, especially nested and composed objects.