Closed rotten closed 3 years ago
Is there a way to share the endpoint? Keeping the structure and anonymize if needed. Seems like a oneOf/anyOf definition.
A bunch of my endpoints do this, only a few actually work. I don't have an easy way to share it at this time. I might be able to construct a standalone FastAPI instance that can reproduce the crash though. It is worth a shot. I'll see what I can post back.
I spun up a simple fastapi example with two GET endpoints that don't really do anything. Now when I run cats both endpoints kick out an NPE. I would have expected the simple endpoints to fuzzable. First the API server: requirements.txt
fastapi==0.65.2
uvicorn==0.14.0
server.py
from fastapi import Depends, FastAPI, Request
app = FastAPI()
@app.get('/')
async def read_root():
return {'Hello': 'World'}
@app.get('/health')
async def api_health(request: Request):
return 'OK'
The api server is started with
uvicorn server:app
You can then hit http://localhost:8000/ and http://localhost:8000/health to see the api response. If you go to http://localhost:8000/docs you can save the "/openapi.json" for use as the contract. I tried it both as a json and converted it to a yml.
./cats.jar --contract=example_openapi.yml --server=http://localhost:8000
I tried specifying the paths individually, and leaving --Paths
out of the argument list.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.4.5)
[********* ][*****] ● note Proxy configuration to be used: DIRECT
[********* ][*****] ▶ start Starting CATS, version 6.0.3-SNAPSHOT, build-time 2021-07-16T15:59:55.285 UTC
[********* ][*****] ● note Processing configuration...
[********* ][*****] ℹ info No security custom Fuzzer file. SecurityFuzzer will be skipped!
[********* ][*****] ℹ info No custom Fuzzer file. CustomFuzzer will be skipped!
[********* ][*****] ℹ info No reference data file was supplied! Payloads supplied by Fuzzers will remain unchanged!
[********* ][*****] ℹ info No URL parameters supplied!
[********* ][*****] ℹ info No headers file was supplied! No additional header will be added!
[********* ][*****] ▶ start skipXXXForPath supplied arguments: []. Matching with registered fuzzers...
[********* ][*****] ☑ complete skipXXXForPath list after matching with registered fuzzers: []
[********* ][*****] ℹ info
[********* ][*****] ℹ info Supplied arguments
[********* ][*****] ℹ info contract: example_openapi.yml
[********* ][*****] ℹ info server: http://localhost:8000
[********* ][*****] ℹ info fuzzers: all
[********* ][*****] ℹ info paths: all
[********* ][*****] ℹ info skipPaths: empty
[********* ][*****] ℹ info excludedFuzzers: empty
[********* ][*****] ℹ info skipXXXForPath: []
[********* ][*****] ℹ info skipFields: empty
[********* ][*****] ℹ info httpMethods: empty
[********* ][*****] ℹ info checkHeaders: false
[********* ][*****] ℹ info checkFields: false
[********* ][*****] ℹ info checkHttp: false
[********* ][*****] ℹ info checkContract: false
[********* ][*****] ℹ info fieldsFuzzingStrategy: ONEBYONE
[********* ][*****] ℹ info maxFieldsToRemove: empty
[********* ][*****] ℹ info edgeSpacesStrategy: trimAndValidate
[********* ][*****] ℹ info sanitizationStrategy: sanitizeAndValidate
[********* ][*****] ℹ info useExamples: true
[********* ][*****] ℹ info reportingLevel: info
[********* ][*****] ℹ info log: empty
[********* ][*****] ℹ info printExecutionStatistics: false
[********* ][*****] ℹ info timestampReports: empty
[********* ][*****] ℹ info reportFormat: htmlJs
[********* ][*****] ℹ info urlParams: empty
[********* ][*****] ℹ info headers: empty
[********* ][*****] ℹ info refData: empty
[********* ][*****] ℹ info customFuzzerFile: empty
[********* ][*****] ℹ info securityFuzzerFile: empty
[********* ][*****] ℹ info proxyPort: 0
[********* ][*****] ℹ info proxyHost: empty
[********* ][*****] ℹ info sslKeystore: empty
[********* ][*****] ℹ info sslKeystorePwd: empty
[********* ][*****] ℹ info sslKeyPwd: empty
[********* ][*****] ℹ info basicauth: empty
[********* ][*****] ☑ complete Finished parsing the contract in 84 ms
[********* ][*****] ⚠ warning
[********* ][*****] ⚠ warning !!!! WARNING !!!!
[********* ][*****] ⚠ warning You are running with all Fuzzers enabled. This will take a long time to run!
[********* ][*****] ⚠ warning Please check the CATS README page for slicing strategies!
[********* ][*****] ⚠ warning
[********* ][*****] ℹ info
[********* ][*****] ▶ start Start fuzzing path /
[********* ][*****] ✖ error Something went wrong while running CATS!
java.lang.NullPointerException: null
at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForNonBodyMethods(FuzzingDataFactory.java:106)
at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzingDataForGet(FuzzingDataFactory.java:87)
at com.endava.cats.model.factory.FuzzingDataFactory.fromPathItem(FuzzingDataFactory.java:72)
at com.endava.cats.CatsMain.fuzzPath(CatsMain.java:360)
at com.endava.cats.CatsMain.startFuzzing(CatsMain.java:174)
at com.endava.cats.CatsMain.doLogic(CatsMain.java:165)
at com.endava.cats.CatsMain.run(CatsMain.java:143)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:819)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:803)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:346)
at com.endava.cats.CatsMain.main(CatsMain.java:102)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
When I get past the NPE, then I can go back to make an endpoint that throws my original exception.
NPE is now fixed. Can you please check. I'm very interested on how the initial payload looks like. It seems something related to the way the swagger python library generates the documentation and how schemas are referenced.
Excellent! Thanks! I can confirm the NPE is gone. Now I'll go back to building my example to see if I can get that IndexOutOfBounds exeception again so we have an example demonstrating the issue. This may take me a couple of days because I have a very full schedule for the next few days.
I believe this is the issue that is causing the IndexOutOfBounds errors: https://github.com/tiangolo/fastapi/issues/240
Is there a way to post an (anonymized) snapshot of the endpoint? I suspect is also around the naming of the schemas?
I'll see what I can do.
I'm still trying to reproduce this, whenever I get a chance, in a fastapi example outside the context of my application. Unfortunately cats works fine on the examples I've been able to generate so far.
I was able to resolve the openapi formatting issue for the "exclusiveMinimum" field as referenced in the issue listed above by exporting a custom openapi.json . (It is still not fixed in fastapi-pagination without customizing the json manually.)
Even the latest version of cats still blows up almost instantly with the stack trace above. So resolving that bug in the openapi.json output did not clear the issue.
Is there any easy way to insert additional debugging hints in the cats code to expose exactly what is causing this exception? I'm not clear whether it is parsing the openapi.json file that is causing it to fail, or something in my endpoint itself. It does not appear to ever actually hit the endpoint.
[********* ][*****] ✖ error Something went wrong while running CATS!
java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 14
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
at java.base/java.lang.String.substring(String.java:1874)
at com.endava.cats.model.factory.FuzzingDataFactory.lambda$addNewCombination$9(FuzzingDataFactory.java:320)
at java.base/java.util.HashMap.forEach(HashMap.java:1336)
at com.endava.cats.model.factory.FuzzingDataFactory.addNewCombination(FuzzingDataFactory.java:318)
at com.endava.cats.model.factory.FuzzingDataFactory.getPayloadCombinationsBasedOnOneOfAndAnyOf(FuzzingDataFactory.java:305)
at com.endava.cats.model.factory.FuzzingDataFactory.generateSample(FuzzingDataFactory.java:289)
at com.endava.cats.model.factory.FuzzingDataFactory.getResponsePayloads(FuzzingDataFactory.java:441)
at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForHttpMethod(FuzzingDataFactory.java:202)
at com.endava.cats.model.factory.FuzzingDataFactory.getFuzzDataForPost(FuzzingDataFactory.java:170)
at com.endava.cats.model.factory.FuzzingDataFactory.fromPathItem(FuzzingDataFactory.java:60)
at com.endava.cats.CatsMain.fuzzPath(CatsMain.java:361)
at com.endava.cats.CatsMain.startFuzzing(CatsMain.java:175)
at com.endava.cats.CatsMain.doLogic(CatsMain.java:166)
at com.endava.cats.CatsMain.run(CatsMain.java:144)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:791)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:775)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:345)
at com.endava.cats.CatsMain.main(CatsMain.java:102)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
I'll keep working on an example whenever I get a chance. I just wanted you to know I haven't given up on it yet.
By adding:
System.out.println(key);
at line 320 of /src/main/java/com/endava/cats/model/factory/FuzzingDataFactory.java
I get these two rows:
dataANY_OF#/components/schemas/response_utils__JsonAPIDocDataItem
dataANY_OFnull
before it blows up.
It looks like the second row is failing because it is missing a #
.
... still digging.
Ok, I found the part of the openapi.json that trips this exception:
In the endpoint, responses, I have a reference to another part of the json:
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/response_utils__JsonAPIDocResult"
}
}
},
"description": "Successful Response"
}
That reference references another object:
"response_utils__JsonAPIDocResult": {
"properties": {
"data": {
"anyOf": [
{
"items": {
"$ref": "#/components/schemas/response_utils__JsonAPIDocDataItem"
},
"type": "array"
},
{
"$ref": "#/components/schemas/response_utils__JsonAPIDocDataItem"
}
],
"title": "Data"
},
The part that breaks is the items
field in the array element. In other words, if I change that anyOf to this:
"response_utils__JsonAPIDocResult": {
"properties": {
"data": {
"anyOf": [
{
"type": "array"
},
{
"$ref": "#/components/schemas/response_utils__JsonAPIDocDataItem"
}
],
"title": "Data"
},
cats doesn't blow up.
(edited because I hit enter too soon)
@rotten Thank you for the detailed information. I'll try to reproduce it locally and work on a fix.
This is now fixed in https://github.com/Endava/cats/commit/59607819ae48ec3ea8ccd564dbd42e0ae6754b90 @rotten please give it a try when you have some time.
Hi Everyone, today I gave a try again and I’m still facing issues. First I got an error about not having the surefire package, then I got this error below. I used the Endava cats maven URL in the Pom file but I also used mine too, I got the same error in both. What should I do? Thank you for your help.
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:30 min
[INFO] Finished at: 2021-08-18T08:42:53-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project cats: Failed to retrieve remote metadata com.endava:cats:6.0.5-SNAPSHOT/maven-metadata.xml: Could not transfer meta
data com.endava:cats:6.0.5-SNAPSHOT/maven-metadata.xml from/to github (https://maven.pkg.github.com/Endava/cats): transfer failed for https://maven.pkg.github.com/Endava/cats/com/endava/cats/6.0.5-SNAPSHOT/maven-metadata.xml:sun.se
curity.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionExceptionhttps://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FMAVEN%2FMojoExecutionException&data=04%7C01%7C%7C8a6483ac4cfd479ec7bf08d962476d27%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648881179685740%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cOZrK1gpZM3s%2BI79OqNYPXJ5gD5%2B7G3mxd%2FDYEjN4CE%3D&reserved=0
Get Outlook for iOShttps://aka.ms/o0ukef
From: Madalin Ilie @.> Sent: Tuesday, August 17, 2021 5:34:10 PM To: Endava/cats @.> Cc: luanacesar @.>; Manual @.> Subject: Re: [Endava/cats] Stack trace on one of my endpoints (#8)
This is now fixed in 5960781https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEndava%2Fcats%2Fcommit%2F59607819ae48ec3ea8ccd564dbd42e0ae6754b90&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523668655%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nDRUodndZDhKtM3cirA4R8tCDG1w%2Bowgye7Z71pf8KA%3D&reserved=0 @rottenhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frotten&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523678650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JEco4Xvp0uWAbgZAKw0KnuD0ldfAiy7jin3Of1qx1L4%3D&reserved=0 please give it a try when you have some time.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEndava%2Fcats%2Fissues%2F8%23issuecomment-900646512&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523678650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BAvtFDrJnOY8tKr0GVhWBPhp7PvIOF7fkqRsBRSZOiU%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPBCYQEEE243RBLP37FOXL3T5LIVFANCNFSM5AQF6BAA&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523688644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZTFN8ETfqexlzkEWAwNXnvgiMY2FZmZgDUBHYCMXkJY%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523688644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yGKPhyv1CDzLC0ZPpHyYiEU6xlFDg8G8tuwxtDdzTDE%3D&reserved=0 or Androidhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7C%7Caa41efc0dead4a5b155308d961c6c0a5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637648328523688644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9yygcIeFpH7hcvkWuOoeM%2FXRe7hyqNQY4b7d%2FJeM2G0%3D&reserved=0.
@luanacesar it seems you are trying to deploy cats. can you please run mvn clean package
and let me know what happens?
This is now released under: https://github.com/Endava/cats/releases/tag/cats-6.0.5 @luanacesar if you still have the other issue reported, please open another issue.
As soon as cats starts up against one of my endpoints, it blows up:
My API does not throw any errors when this happens. Suggestions for what might cause this?