search

Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.2k stars 73 forks source link

cats appears to incorrectly (?) provide a value for a read only field for an unrelated fuzzer #19

Closed aharpervc closed 2 years ago

aharpervc commented 2 years ago

Hi, I've started playing around with cats and noticed something odd. I see a fuzz test report for the RemoveFieldsFuzzer that ran a post request to create a record. Based on the request payload shown in the report, a particular field was actually removed, so that part looks good.

However, another field was included in the request payload that my OpenAPI spec defines as read only (the "id" field in this case). Here's a snippet from the spec, just for context (OpenAPI 3.0.0):

      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                data:
                  type: object
                  properties:
                    id:
                      type: integer
                      description: ID
                      readOnly: true

The request payload was otherwise structured as I'd expect with various dummy/fuzzed values. My expectation in this case is that this fuzzer would isolate the request payload such that fields marked as read only won't be included, so as to isolate the logic and only validate the behavior when some other fields is intentionally excluded.

I'm not sure if this is a cats bug, or a misunderstanding on my part, or something that can be configured. Thoughts?

aharpervc commented 2 years ago

I noticed something else about this particular report... it looks like cats doesn't support http status code "ranges" syntax in I have in the OpenAPI spec file (eg, 4xx as defined here: https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.2.md#patterned-fields-1).

The expected result in the report says "Should return [4XX] response code as required fields [were] removed", and the response code is actually 400. The result details says:

Response does NOT match expected result. Response code is from a list of expected codes for this FUZZER, but it is undocumented: expected [400, 413, 414, 422], actual [400], documented response codes: [200, 4XX, 5XX]".

en-milie commented 2 years ago

Hi, @aharpervc! Thank you for taking the time to report these. They are both valid points. CATS does not exclude readOnly fields and does not support ranges. I'll open a different bug for the ranges one as it is quite simple to enable, and leave this one for the readOnly fields.

Will probably have both available in 7.0.1.

aharpervc commented 2 years ago

Great, I'll look forward to the next release then. It also occurred to me that cats may benefit from a fuzzer that attempts to write to read only fields, and validates that there's an error code as the response.

en-milie commented 2 years ago

@aharpervc Thank you for the idea about a new Fuzzer. The issue should be fixed in https://github.com/Endava/cats/releases/tag/cats-7.0.1