search

Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.19k stars 73 forks source link

Does cats support single interface testing #139

Open duhang1996 opened 2 months ago

duhang1996 commented 2 months ago

Does cats support single interface testing? such as: https://cloud.githubusercontent.com/assets/4562611/19390428/12224610-91f6-11e6-9ece-6e3cd7dd35ea.png

duhang1996 commented 2 months ago

java -jar target/cats-runner.jar fuzz -X GET --mc 500 -t "path1" -i "2XX,4XX" https://petstore3.swagger.io/api/v3/pet/path1

[][] ▶ Starting cats-11.8.1-SNAPSHOT, build time 2024-09-26T06:32:26Z UTC, platform Mac OS X-12.0-aarch64

================================================================================= FUZZING =================================================================================

https://petstore3.swagger.io/api/v3/pet/path1 .............................................................................................................. E 0, W 0, S 0 ✔ [][] ℹ Skip printing time execution statistics. You can use --printExecutionStatistics to enable this feature!

[][] ★ CATS finished in 14.749s. Total requests 0. ✔ Passed 0, ⚠ warnings: 0, ‼ errors: 0, You can open the report here:

Why there were 0 tests generated？Help

en-milie commented 1 month ago

Hi @duhang1996. Yes it does. Exactly as you executed. The report has zero tests because no test matched 500 as you instructed. If you run with --mc 400 you will get all tests in the report. If you want to see the execution details, you can also include --verbosity DETAILED as an argument.

duhang1996 commented 1 month ago

Hi @duhang1996. Yes it does. Exactly as you executed. The report has zero tests because no test matched 500 as you instructed. If you run with --mc 400 you will get all tests in the report. If you want to see the execution details, you can also include --verbosity DETAILED as an argument.

thank you!

duhang1996 commented 1 month ago

Hi @duhang1996. Yes it does. Exactly as you executed. The report has zero tests because no test matched 500 as you instructed. If you run with --mc 400 you will get all tests in the report. If you want to see the execution details, you can also include --verbosity DETAILED as an argument.

Hi, @en-milie After several rounds of testing, I found that the generated test is the same every time. How to keep this test constantly running or generate more test cases？Thanks！

en-milie commented 1 month ago

When you say same tests what do you mean? As a scenario? What CATS does with the built in mechanism is to send a predefined set of random characters in the fields you specify. For more intelligence you should use an OpenAPI spec. That will create more advanced scenarios. You can also provide your own dictionary using --words <file>:

java -jar target/cats-runner.jar fuzz -X GET --mc 400 -t "path1" -i "2XX,4XX" --words nastyStrings.txt https://petstore3.swagger.io/api/v3/pet/path1

duhang1996 commented 1 month ago

@en-milie What I mean is that the random replacement characters generated for the first and second times are exactly the same, and the number of test cases generated is also the same(38).After executing the 38 test cases, the run ends and cannot be tested continuously. the first test:

image image

the second test:

image
en-milie commented 1 month ago

Do you have an OpenAPI spec? Or just fuzzing based on a JSON template?

en-milie commented 2 weeks ago

Starting with this release https://github.com/Endava/cats/releases/tag/cats-13.0.0 you can add an additional --random argument to the cats fuzz sub-command that will do continuous fuzzing.