Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.11k stars 76 forks source link

This PR have fix for two issues mentioned in the description #116

Closed ganeshnikam18 closed 3 months ago

ganeshnikam18 commented 3 months ago
  1. For StringFieldLeftBoundary and StringFieldRightBoundary fuzzer, cats was not fuzzing the fields if minLength or maxLength is not specified for the field. As per the documentation it should fuzz such field with empty string (for leftBoundary) and with 10000 char string (for right boundary). https://endava.github.io/cats/docs/fuzzers/field-fuzzers/string-left-boundary https://endava.github.io/cats/docs/fuzzers/field-fuzzers/string-right-boundary

I have added this fix using the reference of IntegerFieldRightBoundary Fuzzer

  1. I have tested below scenario:
  "scenario": "Send a 'happy' flow request with all fields and all headers",

"result": "warn", "resultReason": "Response content type not matching the contract", "id": "Test 1", "fuzzer": "HappyPath", "path": "/test1/order", "resultDetails": "Response content type not matching the contract: expected [application/json], actual [application/problem+json]", "timeToExecuteInSec": 0.202, "httpMethod": "post",

en-milie commented 3 months ago

For (1) yes, it seems that documentation is not aligned with the implementation. Please also update the corresponding tests as they are failing.

For (2) I am a bit skeptical. Did you use the latest version of CATS to run that test? It shouldn't raise a warn as application/json and application/problem+json content types are equivalent. I agree that we should raise errors first. And it will probably be a better idea to increase the warn in this case to error.

So, in conclusion, please also updates the corresponding tests for (1).

And for (2) try running with latest CATS.

ganeshnikam18 commented 3 months ago

For (1) I have updated the test cases and added the commit

For (2) I have verified with latest version as you suggested. It does not report the warn for content-type mis-match. You can improve the current sequence of check to report error 1st and then warning and then success case. I had added that commit as per my understanding. You can reject the commit

en-milie commented 3 months ago

Merged now, but preserving the existing logic to report results.