Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.2k stars 73 forks source link

CATS fails for HTTP PUT, does not replace the path parameter #1

Closed jonminter closed 4 years ago

jonminter commented 4 years ago

I'm not entirely sure if this is because of the tool not working well with that particular case or if there is something in documentation I missed about configuration options. But what is the expected behavior for HTTP PUT for paths that have a required path parameter in them?

Here is the error I'm receiving when trying to run these tests against my API (full URL path is redacted):

[ERROR] [Test 36] [DuplicateHeaderFuzzer] - Application run failed java.lang.IllegalStateException: Failed to execute CommandLineRunner at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787) at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768) at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) at com.endava.cats.CatsMain.main(CatsMain.java:95) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:597) Caused by: java.lang.IllegalArgumentException: Illegal character in path at index 70: http://localhost:8000/***********************************************/{id} at java.base/java.net.URI.create(URI.java:883) at org.apache.http.client.methods.HttpPut.(HttpPut.java:65) at com.endava.cats.io.ServiceCaller.put(ServiceCaller.java:185) at com.endava.cats.io.ServiceCaller.call(ServiceCaller.java:154) at com.endava.cats.fuzzer.headers.DuplicateHeaderFuzzer.process(DuplicateHeaderFuzzer.java:61) at com.endava.cats.fuzzer.headers.DuplicateHeaderFuzzer.lambda$fuzz$0(DuplicateHeaderFuzzer.java:42) at com.endava.cats.report.TestCaseListener.createAndExecuteTest(TestCaseListener.java:62) at com.endava.cats.fuzzer.headers.DuplicateHeaderFuzzer.fuzz(DuplicateHeaderFuzzer.java:41) at com.endava.cats.fuzzer.headers.DuplicateHeaderFuzzer$$FastClassBySpringCGLIB$$f2dd043b.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.endava.cats.aop.FuzzerLogAspect.logExecutionTime(FuzzerLogAspect.java:19) at jdk.internal.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.endava.cats.fuzzer.headers.DuplicateHeaderFuzzer$$EnhancerBySpringCGLIB$$8ced12e9.fuzz() at com.endava.cats.CatsMain.lambda$fuzzPath$7(CatsMain.java:309) at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) at com.endava.cats.CatsMain.fuzzPath(CatsMain.java:307) at com.endava.cats.CatsMain.startFuzzing(CatsMain.java:152) at com.endava.cats.CatsMain.doLogic(CatsMain.java:146) at com.endava.cats.CatsMain.run(CatsMain.java:125) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784) ... 11 common frames omitted Caused by: java.net.URISyntaxException: Illegal character in path at index 70: http://localhost:8000/***********************************************/{id} at java.base/java.net.URI$Parser.fail(URI.java:2913) at java.base/java.net.URI$Parser.checkChars(URI.java:3084) at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3166) at java.base/java.net.URI$Parser.parse(URI.java:3114) at java.base/java.net.URI.(URI.java:600) at java.base/java.net.URI.create(URI.java:881) ... 45 common frames omitted

en-milie commented 4 years ago

PUT is one of the methods that has an actual body/payload and this is why the fuzzing only happens at the payload level (as opposite to GET, DELETE). Path parameters won't be fuzzed for PUT but rather it is expected to supply a static value via urlParams or the reference data file. In your case the id needs to have a static value. I'll make sure the documentation is more clear about this and will also make the exception to reflect this aspect.

Thank you for submitting this.

jonminter commented 4 years ago

Ok thanks! And no problem, this tool looks great. I'm trying to integrate it into the CI pipeline for a project for automating some API testing should save a lot of time and test a greater range of scenarios than hand written example tests.

en-milie commented 4 years ago

This is now fixed in https://github.com/Endava/cats/releases/tag/cats-4.2.1