search

Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.11k stars 76 forks source link

Update TestCaseExporter.java #97

Closed binaryArrow closed 4 months ago

binaryArrow commented 5 months ago

Catch NegativeArrayizeExcepetion.

I encountered a very confusing NegativeArraySizeException which is not catched by the IOException catch block. Further I observed, that in rare cases the report creation does exceed the java heap size in which case the application crashes. Did a bit of tweaking of the heapsize and garbage collector, but couldn't fix the heap exceeding. I will do further investigations and provide some cases for reproduction by Time. For now catching the exception was the first step for not crashing at least...

en-milie commented 5 months ago

Can you please also add here the stack trace? Is there anything particular about that test case? Does the export fail for a particular fuzzer? For a particular OpenAPI spec? Does it fail both native binaries or the uberjar version of CATS?

binaryArrow commented 5 months ago

In the following is the NegativeArraySizeException which occurs without the catch block in my pull request. With the catch block however the application comes to an halt for about 20 seconds at around 54% of each endpoint which is being fuzzed. The RAM usage grows with every endpoint until the application crashes. This espacially occurs in DEBUG mode, when I start normally, it indeed does sometimes manage to run all the fuzzers and write a report.

I tested the uberjar so I cannot say if it fails on the native binaries. It does not fail for a specific Fuzzer as far as I can say. But it fails for a particular OpenApi spec, with my other specs everything is alright, so it might be something returned by the fuzzed Endpoint which triggers this exception/behavior. If that's the case it could be kinda challenging to provide a straightforward testcase. I will investigate further and try to create a Testcase for reproduction..

java.lang.NegativeArraySizeException: -1880557586
        at java.base/java.lang.String.encodeUTF8(String.java:1270)
        at java.base/java.lang.String.getBytesNoRepl1(String.java:933)
        at java.base/java.lang.String.getBytesNoRepl(String.java:915)
        at java.base/java.lang.System$2.getBytesNoRepl(System.java:2394)
        at java.base/java.nio.file.Files.writeString(Files.java:3731)
        at java.base/java.nio.file.Files.writeString(Files.java:3674)
        at com.endava.cats.report.TestCaseExporter.writeHtmlTestCase(TestCaseExporter.java:312)
        at com.endava.cats.report.TestCaseExporter.writeTestCase(TestCaseExporter.java:283)
        at com.endava.cats.report.TestCaseExporterHtmlJs_Subclass.writeTestCase$$superforward(Unknown Source)
        at com.endava.cats.report.TestCaseExporterHtmlJs_Subclass$$function$$1.apply(Unknown Source)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
        at com.endava.cats.aop.DryRunAspect.intercept(DryRunAspect.java:134)
        at com.endava.cats.aop.DryRunAspect_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
        at com.endava.cats.report.TestCaseExporterHtmlJs_Subclass.writeTestCase(Unknown Source)
        at com.endava.cats.report.TestCaseListener.endTestCase(TestCaseListener.java:183)
        at com.endava.cats.report.TestCaseListener.createAndExecuteTest(TestCaseListener.java:129)
        at com.endava.cats.report.TestCaseListener_Subclass.createAndExecuteTest$$superforward(Unknown Source)
        at com.endava.cats.report.TestCaseListener_Subclass$$function$$13.apply(Unknown Source)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
        at com.endava.cats.aop.DryRunAspect.intercept(DryRunAspect.java:134)
        at com.endava.cats.aop.DryRunAspect_Bean.intercept(Unknown Source)
        at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
        at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
        at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
        at com.endava.cats.report.TestCaseListener_Subclass.createAndExecuteTest(Unknown Source)
        at com.endava.cats.report.TestCaseListener_ClientProxy.createAndExecuteTest(Unknown Source)
        at com.endava.cats.fuzzer.executor.FieldsIteratorExecutor.execute(FieldsIteratorExecutor.java:74)
        at com.endava.cats.fuzzer.fields.base.BaseReplaceFieldsFuzzer.fuzz(BaseReplaceFieldsFuzzer.java:30)
        at com.endava.cats.command.CatsCommand.lambda$runFuzzers$8(CatsCommand.java:364)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at com.endava.cats.command.CatsCommand.runFuzzers(CatsCommand.java:360)
        at com.endava.cats.command.CatsCommand.fuzzPath(CatsCommand.java:350)
        at com.endava.cats.command.CatsCommand.startFuzzing(CatsCommand.java:264)
        at com.endava.cats.command.CatsCommand.doLogic(CatsCommand.java:204)
        at com.endava.cats.command.CatsCommand.run(CatsCommand.java:156)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
        at picocli.CommandLine.execute(CommandLine.java:2170)
        at com.endava.cats.CatsMain.run(CatsMain.java:39)
        at com.endava.cats.CatsMain_ClientProxy.run(Unknown Source)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:132)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
        at java.base/java.lang.Thread.run(Thread.java:833)
en-milie commented 5 months ago

I suppose you cannot post a snapshot of the OpenAPI spec here. Or some anonymised one.

binaryArrow commented 5 months ago

Unfortunetly I cannot post it and it's a rather big one to do a quick anonymization.

en-milie commented 5 months ago

Do you run this on Windows? If not, can you please also try with the native binaries?

binaryArrow commented 5 months ago

With the Binaries I get this Exception:

But i fixed that in my pull request erlier, does the current binaries have these changes?

 java.lang.IllegalArgumentException: Length cannot be greater than 14
    at org.cornutum.regexpgen.Bounds.clippedTo(Bounds.java:67)
    at org.cornutum.regexpgen.RegExpGen.effectiveLength(RegExpGen.java:89)
    at org.cornutum.regexpgen.js.AbstractRegExpGen.generate(AbstractRegExpGen.java:102)
    at org.cornutum.regexpgen.RegExpGen.generate(RegExpGen.java:61)
    at com.endava.cats.generator.simple.StringGenerator.generateUsingRegexpGen(StringGenerator.java:124)
    at com.endava.cats.generator.simple.StringGenerator.generate(StringGenerator.java:116)
    at com.endava.cats.generator.simple.StringGenerator.generateExactLength(StringGenerator.java:83)
    at com.endava.cats.fuzzer.fields.base.ExactValuesInFieldsFuzzer.getBoundaryValue(ExactValuesInFieldsFuzzer.java:65)
    at com.endava.cats.fuzzer.fields.base.ExactValuesInFieldsFuzzer.getBoundaryValue(ExactValuesInFieldsFuzzer.java:21)
    at com.endava.cats.fuzzer.fields.base.BaseBoundaryFieldFuzzer.fuzzerGeneratedBoundaryValue(BaseBoundaryFieldFuzzer.java:65)
    at com.endava.cats.fuzzer.fields.base.BaseBoundaryFieldFuzzer.getFieldFuzzingStrategy(BaseBoundaryFieldFuzzer.java:37)
    at com.endava.cats.fuzzer.fields.base.BaseFieldsFuzzer.fuzz(BaseFieldsFuzzer.java:63)
    at com.endava.cats.command.CatsCommand.lambda$runFuzzers$8(CatsCommand.java:364)
    at java.base@20.0.2/java.util.ArrayList.forEach(ArrayList.java:1511)
    at com.endava.cats.command.CatsCommand.runFuzzers(CatsCommand.java:360)
    at com.endava.cats.command.CatsCommand.fuzzPath(CatsCommand.java:350)
    at com.endava.cats.command.CatsCommand.startFuzzing(CatsCommand.java:264)
    at com.endava.cats.command.CatsCommand.doLogic(CatsCommand.java:204)
    at com.endava.cats.command.CatsCommand.run(CatsCommand.java:156)
    at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
    at picocli.CommandLine.access$1500(CommandLine.java:148)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
    at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
    at picocli.CommandLine.execute(CommandLine.java:2170)
    at com.endava.cats.CatsMain.run(CatsMain.java:39)
    at com.endava.cats.CatsMain_ClientProxy.run(Unknown Source)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:132)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runner.GeneratedMain.main(Unknown Source)
en-milie commented 5 months ago

Thanks! No, not yet. It will be in the next release, today or tomorrow.

binaryArrow commented 5 months ago

okay, I will try it with the binaries in the next release then.

en-milie commented 5 months ago

Can you please try with the latest binary: https://github.com/Endava/cats/releases/tag/cats-10.5.0

binaryArrow commented 5 months ago

Hey, I will on Wednesday or Thursday.

en-milie commented 4 months ago

Hi @binaryArrow. Better luck with the latest native binary?

binaryArrow commented 4 months ago

I am still sick so couldn't work. I will do it the coming week, I'm really sorry. I let you know=)

en-milie commented 4 months ago

No worries at all! Health comes first. Take care!

binaryArrow commented 4 months ago

Hey! So I finally tested it with the native binary and unfortunately I still get the same Exception. It might be caused by some response by our API but I have to investigate more in that direction... Here's the Exception ( almost the same, but slightly different numbers)

java.lang.NegativeArraySizeException: -1880877808
    at java.base@20.0.2/java.lang.String.encodeUTF8(String.java:1288)
    at java.base@20.0.2/java.lang.String.getBytesNoRepl1(String.java:953)
    at java.base@20.0.2/java.lang.String.getBytesNoRepl(String.java:935)
    at java.base@20.0.2/java.lang.System$2.getBytesNoRepl(System.java:2469)
    at java.base@20.0.2/java.nio.file.Files.writeString(Files.java:3706)
    at java.base@20.0.2/java.nio.file.Files.writeString(Files.java:3649)
    at com.endava.cats.report.TestCaseExporter.writeHtmlTestCase(TestCaseExporter.java:356)
    at com.endava.cats.report.TestCaseExporter.writeTestCase(TestCaseExporter.java:329)
    at com.endava.cats.report.TestCaseExporterHtmlJs_Subclass.writeTestCase$$superforward(Unknown Source)
    at com.endava.cats.report.TestCaseExporterHtmlJs_Subclass$$function$$1.apply(Unknown Source)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
    at com.endava.cats.aop.DryRunAspect.intercept(DryRunAspect.java:157)
    at com.endava.cats.aop.DryRunAspect_Bean.intercept(Unknown Source)
    at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
    at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
    at com.endava.cats.report.TestCaseExporterHtmlJs_Subclass.writeTestCase(Unknown Source)
    at com.endava.cats.report.TestCaseListener.endTestCase(TestCaseListener.java:269)
    at com.endava.cats.report.TestCaseListener.createAndExecuteTest(TestCaseListener.java:159)
    at com.endava.cats.report.TestCaseListener_Subclass.createAndExecuteTest$$superforward(Unknown Source)
    at com.endava.cats.report.TestCaseListener_Subclass$$function$$13.apply(Unknown Source)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
    at com.endava.cats.aop.DryRunAspect.intercept(DryRunAspect.java:157)
    at com.endava.cats.aop.DryRunAspect_Bean.intercept(Unknown Source)
    at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
    at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
    at com.endava.cats.report.TestCaseListener_Subclass.createAndExecuteTest(Unknown Source)
    at com.endava.cats.report.TestCaseListener_ClientProxy.createAndExecuteTest(Unknown Source)
    at com.endava.cats.fuzzer.executor.FieldsIteratorExecutor.execute(FieldsIteratorExecutor.java:97)
    at com.endava.cats.fuzzer.fields.base.BaseReplaceFieldsFuzzer.fuzz(BaseReplaceFieldsFuzzer.java:42)
    at com.endava.cats.command.CatsCommand.lambda$runFuzzers$8(CatsCommand.java:391)
    at java.base@20.0.2/java.util.ArrayList.forEach(ArrayList.java:1511)
    at com.endava.cats.command.CatsCommand.runFuzzers(CatsCommand.java:387)
    at com.endava.cats.command.CatsCommand.fuzzPath(CatsCommand.java:373)
    at com.endava.cats.command.CatsCommand.startFuzzing(CatsCommand.java:286)
    at com.endava.cats.command.CatsCommand.doLogic(CatsCommand.java:226)
    at com.endava.cats.command.CatsCommand.run(CatsCommand.java:177)
    at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
    at picocli.CommandLine.access$1500(CommandLine.java:148)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
    at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
    at picocli.CommandLine.execute(CommandLine.java:2170)
    at com.endava.cats.CatsMain.run(CatsMain.java:42)
    at com.endava.cats.CatsMain_ClientProxy.run(Unknown Source)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:132)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
    at io.quarkus.runner.GeneratedMain.main(Unknown Source)
en-milie commented 4 months ago

Ok. I'll merge the PR so that it doesn't fail and maybe we get more details after. Might be something with the response encoding.