CucumberOptions annotation causes exception in 0.6.2

[msennett9]

I have a set of tests that works fine with Karate 0.6.1. In the top-level test runner class, we have the following annotation to support Cucumber reporting: @CucumberOptions(plugin = {"pretty", "json:target/cucumber/cucumber-data"}). When I try to upgrade to 0.6.2, this results in all tests failing, reporting that all scenarios have 0 scenarios and 0 steps. To recreate, generate a project with the archetype, After running to make sure it works, add the annotation mentioned above (along with an import import cucumber.api.CucumberOptions; to ExamplesTest.java. Run mvn test to see the failure. The first exception:

-------------------------------------------------------------------------------
Test set: examples.ExamplesTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.358 sec <<< FAILURE!
Feature: sample karate test script  Time elapsed: 0.765 sec  <<< ERROR!
java.lang.NullPointerException
        at gherkin.formatter.JSONFormatter.match(JSONFormatter.java:123)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at cucumber.runtime.Utils$1.call(Utils.java:40)
        at cucumber.runtime.Timeout.timeout(Timeout.java:16)
        at cucumber.runtime.Utils.invoke(Utils.java:34)
        at cucumber.runtime.RuntimeOptions$1.invoke(RuntimeOptions.java:294)
        at com.sun.proxy.$Proxy6.match(Unknown Source)
        at com.intuit.karate.cucumber.KarateHtmlReporter.match(KarateHtmlReporter.java:307)
        at cucumber.runtime.junit.JUnitReporter.match(JUnitReporter.java:68)
        at com.intuit.karate.junit4.Karate$1.match(Karate.java:90)
        at com.intuit.karate.cucumber.CucumberUtils.afterStep(CucumberUtils.java:160)
        at com.intuit.karate.cucumber.CucumberUtils.runStep(CucumberUtils.java:146)
        at com.intuit.karate.cucumber.KarateRuntime.runStep(KarateRuntime.java:80)
        at cucumber.runtime.model.StepContainer.runStep(StepContainer.java:44)
        at cucumber.runtime.model.StepContainer.runSteps(StepContainer.java:39)
        at cucumber.runtime.model.CucumberScenario.runBackground(CucumberScenario.java:59)
        at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:42)
        at cucumber.runtime.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:102)
        at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:63)
        at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:18)
        ...

Changing the Karate version back to 0.6.1 and the example will work again.

[ptrthomas]

@msennett9 I'll look at this in a moment. meanwhile, I'd like to explore if you would be okay to NOT use the pretty and json 'native' reports.

especially the json, in Karate you are supposed to use what is generated by the parallel runner, refer: https://github.com/intuit/karate#parallel-execution

and pretty was useful only in dev mode and now you have the HTML report: https://github.com/intuit/karate#junit-html-report

I may have missed making this clear in the doc, and instead of a null-pointer I'm considering a helpful error message instead. would that work ?

[msennett9]

I did see the doc about the parallel runner. We are generating the json because we wanted to use the net.masterthought maven plugin for generating the nice-looking Cucumber reports. I'll take a look to see if we can use another approach.

[ptrthomas]

@msennett9 the karate json is standard with some enhancements (most importantly the in-line HTTP logs, plus some nice tweaks for api testing, for e.g. you see the feature file name not the 'friendly' name) and should work with the masterthought plugin.

[ptrthomas]

@msennett9 were you able to try the karate parallel runner + json + the masterthought reports ?

also, I released an un-official 0.6.2.3 version, would be great if you try and let me know if fixed.

[msennett9]

I just tried the 0.7.0.RC6 version and it doesn't fail in the same way, but instead it gets through one scenario in the first feature file and then seems to get confused and stops parsing the Karate commands correctly at all. It ends up with errors like this:

[ERROR] Feature: Basic Checkout  Time elapsed: 0 s  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: 20

[ERROR] Feature: Basic Checkout Using GET  Time elapsed: 0 s  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: -1

[ERROR] Feature: Test Split Zone  Time elapsed: 0 s  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: 32

[ERROR] Feature: County Validation  Time elapsed: 0 s  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: -2

[ERROR] Feature: PPC Reporting Validation  Time elapsed: 0 s  <<< ERROR!
java.lang.StringIndexOutOfBoundsException: String index out of range: -107

I haven't had time to try the recommended approach yet, but will try to do so soon. However, I don't think this is really fixed.

[ptrthomas]

considering as still open. will look at nuclear option of failing build if using un-supported combination of plugins

[ptrthomas]

fixed in dev. now any cucumber plugins specified will be completely ignored

[msennett9]

I am now using the cucumber json generated by the parallel runner. It appears to work fine in our use case - both 0.6.2 and 0.7.0.RC6.