Strange results number (always 0 successful, additional 2 errors)

[qarmin]

Describe the bug When running fuzzer I see that it shows that 2 requests were done and I see 2 errors, but in this run I at the right side of endpoints I see 0 errors, 0 warnings and 0 successfull tests(there was more endpoints, but all with identical 0 values)

In tested app logs I see hundreads of successful and unsuccessful requests - so looks that -b option always returns 0, but should normally count this operations as successfull(removing -b cause to show normal amount of successfull events)

./cats --server=http://127.0.0.1:5000 --contract=/home/rafal/Pobrane/openapi.json -b -k --skipFuzzers=NonRestHttpMethodsFuzzer,LargeNumberOfRandomAlphanumericHeadersFuzzer,LargeNumberOfRandomHeadersFuzzer,VeryLargeUnicodeStringsInFieldsFuzzer

[******][*******] ▶ Starting cats-10.1.0, build time 2023-11-21T06:39:04Z UTC, platform Linux-6.2.0-36-generic-amd64 
[******][*******] ⚙ OpenAPI specs: /home/rafal/Pobrane/openapi.json 
[******][*******] ⚙ API base url: http://127.0.0.1:5000 
[******][*******] ⚙ Reporting path: cats-report 
[******][*******] ⚙ 84 configured fuzzers out of 128 total fuzzers 
[******][*******] ⚙ 28 configured paths out of 28 total OpenAPI paths 
[******][*******] ⚙ HTTP methods in scope: [POST, PUT, GET, TRACE, DELETE, PATCH, HEAD] 
[******][*******] ⚙ Total number of OpenAPI operations: 34 

=================================================================================== FUZZING ===================================================================================

/rest/1/backup............................................................................................................................................. E 0, W 0, S 0  100%
/rest/1/backup/............................................................................................................................................ E 0, W 0, S 0  100%
[******][*******] ℹ Skip printing time execution statistics. You can use --printExecutionStatistics to enable this feature! 

[******][*******] ★ CATS finished in 2m59.261s. Total requests 2. ✔ Passed 0, ⚠ warnings: 0, ‼ errors: 2, You can open the report here: file:///home/rafal/Pobrane/cats_linux_amd64_10.1.0/cats-report/index.html 

When looking at 2 failed tests I see that tests were created by "CheckDeletedResourcesNotAvailable" fuzzer.

[en-milie]

Hi @qarmin. There is indeed a small issue, due to the fact the CheckDeletedResourcesNotAvailable fails due a path parsing issue (which will be fixed in the next release), but I also would like to clarify how those 2 arguments are working:

• -b or --blackbox means that CATS will ignore all errors reported by its fuzzers as long as the response code received is 2xx and 4xx (and soon also 501). Which means that errors are reported for 5xx. If you run just with -b but without -k you will all tests being reported
• -k or --skipReportingForIgnoredCodes will skip reporting for any http response code supplied by -b or through the dedicated --ignoreResponseCodes argument. Which means that there won't be any reporting done for those skipped codes, and only report if there are any errors.

So if you run with both -b and -k CATS will ignore all non-5xx responses and won't do any reporting unless your service will actually return a 500 for example.

[en-milie]

I'll close this for now. Please let me know @qarmin if something is not clear.