Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.17k stars 74 forks source link

Stack trace on one of my endpoints #8

Closed rotten closed 3 years ago

rotten commented 3 years ago

As soon as cats starts up against one of my endpoints, it blows up:

[********* ][*****] ▶ start    Start fuzzing path /v1/my/endpointl
[********* ][*****] ✖ error    Something went wrong while running CATS!
java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 14
    at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
    at java.base/java.lang.String.substring(String.java:1874)
    at com.endava.cats.model.factory.FuzzingDataFactory.lambda$addNewCombination$9(FuzzingDataFactory.java:320)
    at java.base/java.util.HashMap.forEach(HashMap.java:1336)
    at com.endava.cats.model.factory.FuzzingDataFactory.addNewCombination(FuzzingDataFactory.java:318)
    at com.endava.cats.model.factory.FuzzingDataFactory.getPayloadCombinationsBasedOnOneOfAndAnyOf(FuzzingDataFactory.java:305)
    at com.endava.cats.model.factory.FuzzingDataFactory.generateSample(FuzzingDataFactory.java:289)
    at com.endava.cats.model.factory.FuzzingDataFactory.getResponsePayloads(FuzzingDataFactory.java:441)
    at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForNonBodyMethods(FuzzingDataFactory.java:110)
    at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzingDataForGet(FuzzingDataFactory.java:87)
    at com.endava.cats.model.factory.FuzzingDataFactory.fromPathItem(FuzzingDataFactory.java:72)
    at com.endava.cats.CatsMain.fuzzPath(CatsMain.java:360)
    at com.endava.cats.CatsMain.startFuzzing(CatsMain.java:174)
    at com.endava.cats.CatsMain.doLogic(CatsMain.java:165)
    at com.endava.cats.CatsMain.run(CatsMain.java:143)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:819)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:803)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:346)
    at com.endava.cats.CatsMain.main(CatsMain.java:102)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)

My API does not throw any errors when this happens. Suggestions for what might cause this?

en-milie commented 3 years ago

Is there a way to share the endpoint? Keeping the structure and anonymize if needed. Seems like a oneOf/anyOf definition.

rotten commented 3 years ago

A bunch of my endpoints do this, only a few actually work. I don't have an easy way to share it at this time. I might be able to construct a standalone FastAPI instance that can reproduce the crash though. It is worth a shot. I'll see what I can post back.

rotten commented 3 years ago

I spun up a simple fastapi example with two GET endpoints that don't really do anything. Now when I run cats both endpoints kick out an NPE. I would have expected the simple endpoints to fuzzable. First the API server: requirements.txt

fastapi==0.65.2
uvicorn==0.14.0

server.py

from fastapi import Depends, FastAPI, Request

app = FastAPI()

@app.get('/')
async def read_root():
    return {'Hello': 'World'}

@app.get('/health')
async def api_health(request: Request):
    return 'OK'

The api server is started with

 uvicorn server:app

You can then hit http://localhost:8000/ and http://localhost:8000/health to see the api response. If you go to http://localhost:8000/docs you can save the "/openapi.json" for use as the contract. I tried it both as a json and converted it to a yml.

 ./cats.jar --contract=example_openapi.yml --server=http://localhost:8000

I tried specifying the paths individually, and leaving --Paths out of the argument list.

rotten commented 3 years ago

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.4.5)

[********* ][*****] ● note     Proxy configuration to be used: DIRECT
[********* ][*****] ▶ start    Starting CATS, version 6.0.3-SNAPSHOT, build-time 2021-07-16T15:59:55.285 UTC
[********* ][*****] ● note     Processing configuration...
[********* ][*****] ℹ info     No security custom Fuzzer file. SecurityFuzzer will be skipped!
[********* ][*****] ℹ info     No custom Fuzzer file. CustomFuzzer will be skipped!
[********* ][*****] ℹ info     No reference data file was supplied! Payloads supplied by Fuzzers will remain unchanged!
[********* ][*****] ℹ info     No URL parameters supplied!
[********* ][*****] ℹ info     No headers file was supplied! No additional header will be added!
[********* ][*****] ▶ start    skipXXXForPath supplied arguments: []. Matching with registered fuzzers...
[********* ][*****] ☑ complete skipXXXForPath list after matching with registered fuzzers: []
[********* ][*****] ℹ info
[********* ][*****] ℹ info     Supplied arguments
[********* ][*****] ℹ info     contract: example_openapi.yml
[********* ][*****] ℹ info     server: http://localhost:8000
[********* ][*****] ℹ info     fuzzers: all
[********* ][*****] ℹ info     paths: all
[********* ][*****] ℹ info     skipPaths: empty
[********* ][*****] ℹ info     excludedFuzzers: empty
[********* ][*****] ℹ info     skipXXXForPath: []
[********* ][*****] ℹ info     skipFields: empty
[********* ][*****] ℹ info     httpMethods: empty
[********* ][*****] ℹ info     checkHeaders: false
[********* ][*****] ℹ info     checkFields: false
[********* ][*****] ℹ info     checkHttp: false
[********* ][*****] ℹ info     checkContract: false
[********* ][*****] ℹ info     fieldsFuzzingStrategy: ONEBYONE
[********* ][*****] ℹ info     maxFieldsToRemove: empty
[********* ][*****] ℹ info     edgeSpacesStrategy: trimAndValidate
[********* ][*****] ℹ info     sanitizationStrategy: sanitizeAndValidate
[********* ][*****] ℹ info     useExamples: true
[********* ][*****] ℹ info     reportingLevel: info
[********* ][*****] ℹ info     log: empty
[********* ][*****] ℹ info     printExecutionStatistics: false
[********* ][*****] ℹ info     timestampReports: empty
[********* ][*****] ℹ info     reportFormat: htmlJs
[********* ][*****] ℹ info     urlParams: empty
[********* ][*****] ℹ info     headers: empty
[********* ][*****] ℹ info     refData: empty
[********* ][*****] ℹ info     customFuzzerFile: empty
[********* ][*****] ℹ info     securityFuzzerFile: empty
[********* ][*****] ℹ info     proxyPort: 0
[********* ][*****] ℹ info     proxyHost: empty
[********* ][*****] ℹ info     sslKeystore: empty
[********* ][*****] ℹ info     sslKeystorePwd: empty
[********* ][*****] ℹ info     sslKeyPwd: empty
[********* ][*****] ℹ info     basicauth: empty
[********* ][*****] ☑ complete Finished parsing the contract in 84 ms
[********* ][*****] ⚠ warning
[********* ][*****] ⚠ warning            !!!! WARNING !!!!
[********* ][*****] ⚠ warning You are running with all Fuzzers enabled. This will take a long time to run!
[********* ][*****] ⚠ warning        Please check the CATS README page for slicing strategies!
[********* ][*****] ⚠ warning
[********* ][*****] ℹ info
[********* ][*****] ▶ start    Start fuzzing path /
[********* ][*****] ✖ error    Something went wrong while running CATS!
java.lang.NullPointerException: null
    at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForNonBodyMethods(FuzzingDataFactory.java:106)
    at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzingDataForGet(FuzzingDataFactory.java:87)
    at com.endava.cats.model.factory.FuzzingDataFactory.fromPathItem(FuzzingDataFactory.java:72)
    at com.endava.cats.CatsMain.fuzzPath(CatsMain.java:360)
    at com.endava.cats.CatsMain.startFuzzing(CatsMain.java:174)
    at com.endava.cats.CatsMain.doLogic(CatsMain.java:165)
    at com.endava.cats.CatsMain.run(CatsMain.java:143)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:819)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:803)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:346)
    at com.endava.cats.CatsMain.main(CatsMain.java:102)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)

When I get past the NPE, then I can go back to make an endpoint that throws my original exception.

en-milie commented 3 years ago

NPE is now fixed. Can you please check. I'm very interested on how the initial payload looks like. It seems something related to the way the swagger python library generates the documentation and how schemas are referenced.

rotten commented 3 years ago

Excellent! Thanks! I can confirm the NPE is gone. Now I'll go back to building my example to see if I can get that IndexOutOfBounds exeception again so we have an example demonstrating the issue. This may take me a couple of days because I have a very full schedule for the next few days.

rotten commented 3 years ago

I believe this is the issue that is causing the IndexOutOfBounds errors: https://github.com/tiangolo/fastapi/issues/240

en-milie commented 3 years ago

Is there a way to post an (anonymized) snapshot of the endpoint? I suspect is also around the naming of the schemas?

rotten commented 3 years ago

I'll see what I can do.

rotten commented 3 years ago

I'm still trying to reproduce this, whenever I get a chance, in a fastapi example outside the context of my application. Unfortunately cats works fine on the examples I've been able to generate so far.

I was able to resolve the openapi formatting issue for the "exclusiveMinimum" field as referenced in the issue listed above by exporting a custom openapi.json . (It is still not fixed in fastapi-pagination without customizing the json manually.)

Even the latest version of cats still blows up almost instantly with the stack trace above. So resolving that bug in the openapi.json output did not clear the issue.

Is there any easy way to insert additional debugging hints in the cats code to expose exactly what is causing this exception? I'm not clear whether it is parsing the openapi.json file that is causing it to fail, or something in my endpoint itself. It does not appear to ever actually hit the endpoint.

[********* ][*****] ✖ error    Something went wrong while running CATS!
java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 14
    at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
    at java.base/java.lang.String.substring(String.java:1874)
    at com.endava.cats.model.factory.FuzzingDataFactory.lambda$addNewCombination$9(FuzzingDataFactory.java:320)
    at java.base/java.util.HashMap.forEach(HashMap.java:1336)
    at com.endava.cats.model.factory.FuzzingDataFactory.addNewCombination(FuzzingDataFactory.java:318)
    at com.endava.cats.model.factory.FuzzingDataFactory.getPayloadCombinationsBasedOnOneOfAndAnyOf(FuzzingDataFactory.java:305)
    at com.endava.cats.model.factory.FuzzingDataFactory.generateSample(FuzzingDataFactory.java:289)
    at com.endava.cats.model.factory.FuzzingDataFactory.getResponsePayloads(FuzzingDataFactory.java:441)
    at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForHttpMethod(FuzzingDataFactory.java:202)
    at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForPost(FuzzingDataFactory.java:170)
    at com.endava.cats.model.factory.FuzzingDataFactory.fromPathItem(FuzzingDataFactory.java:60)
    at com.endava.cats.CatsMain.fuzzPath(CatsMain.java:361)
    at com.endava.cats.CatsMain.startFuzzing(CatsMain.java:175)
    at com.endava.cats.CatsMain.doLogic(CatsMain.java:166)
    at com.endava.cats.CatsMain.run(CatsMain.java:144)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:791)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:775)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:345)
    at com.endava.cats.CatsMain.main(CatsMain.java:102)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)

I'll keep working on an example whenever I get a chance. I just wanted you to know I haven't given up on it yet.

rotten commented 3 years ago

By adding:

System.out.println(key);

at line 320 of /src/main/java/com/endava/cats/model/factory/FuzzingDataFactory.java

I get these two rows:

dataANY_OF#/components/schemas/response_utils__JsonAPIDocDataItem
dataANY_OFnull

before it blows up.

It looks like the second row is failing because it is missing a #.

... still digging.

rotten commented 3 years ago

Ok, I found the part of the openapi.json that trips this exception:

In the endpoint, responses, I have a reference to another part of the json:

                "responses": {
                    "200": {
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/response_utils__JsonAPIDocResult"
                                }
                            }
                        },
                        "description": "Successful Response"
                    }

That reference references another object:

            "response_utils__JsonAPIDocResult": {
                "properties": {
                    "data": {
                        "anyOf": [
                            {
                                "items": {
                                    "$ref": "#/components/schemas/response_utils__JsonAPIDocDataItem"
                                },
                                "type": "array"
                            },
                            {
                                "$ref": "#/components/schemas/response_utils__JsonAPIDocDataItem"
                            }
                        ],
                        "title": "Data"
                    },

The part that breaks is the items field in the array element. In other words, if I change that anyOf to this:

            "response_utils__JsonAPIDocResult": {
                "properties": {
                    "data": {
                        "anyOf": [
                            {
                                "type": "array"
                            },
                            {
                                "$ref": "#/components/schemas/response_utils__JsonAPIDocDataItem"
                            }
                        ],
                        "title": "Data"
                    },

cats doesn't blow up.

(edited because I hit enter too soon)

en-milie commented 3 years ago

@rotten Thank you for the detailed information. I'll try to reproduce it locally and work on a fix.

en-milie commented 3 years ago

This is now fixed in https://github.com/Endava/cats/commit/59607819ae48ec3ea8ccd564dbd42e0ae6754b90 @rotten please give it a try when you have some time.

luanacesar commented 3 years ago

Hi Everyone, today I gave a try again and I’m still facing issues. First I got an error about not having the surefire package, then I got this error below. I used the Endava cats maven URL in the Pom file but I also used mine too, I got the same error in both. What should I do? Thank you for your help.

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 02:30 min

[INFO] Finished at: 2021-08-18T08:42:53-04:00

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project cats: Failed to retrieve remote metadata com.endava:cats:6.0.5-SNAPSHOT/maven-metadata.xml: Could not transfer meta

data com.endava:cats:6.0.5-SNAPSHOT/maven-metadata.xml from/to github (https://maven.pkg.github.com/Endava/cats): transfer failed for https://maven.pkg.github.com/Endava/cats/com/endava/cats/6.0.5-SNAPSHOT/maven-metadata.xml:sun.se

curity.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionExceptionhttps://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FMAVEN%2FMojoExecutionException&data=04%7C01%7C%7C8a6483ac4cfd479ec7bf08d962476d27%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648881179685740%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cOZrK1gpZM3s%2BI79OqNYPXJ5gD5%2B7G3mxd%2FDYEjN4CE%3D&reserved=0


org.apache.maven.plugins maven-surefire-plugin true

Get Outlook for iOShttps://aka.ms/o0ukef


From: Madalin Ilie @.> Sent: Tuesday, August 17, 2021 5:34:10 PM To: Endava/cats @.> Cc: luanacesar @.>; Manual @.> Subject: Re: [Endava/cats] Stack trace on one of my endpoints (#8)

This is now fixed in 5960781https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEndava%2Fcats%2Fcommit%2F59607819ae48ec3ea8ccd564dbd42e0ae6754b90&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523668655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nDRUodndZDhKtM3cirA4R8tCDG1w%2Bowgye7Z71pf8KA%3D&reserved=0 @rottenhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frotten&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523678650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JEco4Xvp0uWAbgZAKw0KnuD0ldfAiy7jin3Of1qx1L4%3D&reserved=0 please give it a try when you have some time.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEndava%2Fcats%2Fissues%2F8%23issuecomment-900646512&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523678650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BAvtFDrJnOY8tKr0GVhWBPhp7PvIOF7fkqRsBRSZOiU%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPBCYQEEE243RBLP37FOXL3T5LIVFANCNFSM5AQF6BAA&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523688644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZTFN8ETfqexlzkEWAwNXnvgiMY2FZmZgDUBHYCMXkJY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523688644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yGKPhyv1CDzLC0ZPpHyYiEU6xlFDg8G8tuwxtDdzTDE%3D&reserved=0 or Androidhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523688644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9yygcIeFpH7hcvkWuOoeM%2FXRe7hyqNQY4b7d%2FJeM2G0%3D&reserved=0.

en-milie commented 3 years ago

@luanacesar it seems you are trying to deploy cats. can you please run mvn clean package and let me know what happens?

en-milie commented 3 years ago

This is now released under: https://github.com/Endava/cats/releases/tag/cats-6.0.5 @luanacesar if you still have the other issue reported, please open another issue.