runLevel in courgette runners problems

[mykpatryk]

Hi, In our project we are trying to introduce xRay plugin to our JIRA, but we encountered on some issue associated with difference in courgette report.json format between set courgette runnerrunLevel = CourgetteRunLevel.FEATUREand runLevel = CourgetteRunLevel.SCENARIO.

While using runLevel = CourgetteRunLevel.SCENARIO xRay plugin cannot import successfully results to xRay because json format provided by courgette mismatched with the expected one. Why? Generally, the problem occurs when we have Examples section in our Scenario Outline. When we have set runLevel = CourgetteRunLevel.SCENARIO generated courgette report json format contains one list of executed examples but when we set runLevel = CourgetteRunLevel.FEATURE courgette report json format looks like every single Example is represented as separated list of only 1 example what provides errors while importing results to xRay. It that an expected behaviour?

Because of the issue which I describe above, I tried to set runLevel = CourgetteRunLevel.FEATURE (because that json format is accepted by xRay ) but it provides us some let say unexpected behaviour on our side while re-running FAILED scenarios. Let me show you situation as example: We have 2 Scenarios in 1 Feature, When one of them is PASSED and the second one is FAILED re-run will executed again both of scenarios irregardless that one of them was already PASSED what provides us 1. Overlong execution time 2. Not clear results as some times happens when first run is PASSED but while re-run it unfortunately fails.

Below I attatched screenshots with:

• correct json:

• wrong json

[prashant-ramcharan]

Hi,

The short answer: The Cucumber json that Courgette generates when using CourgetteRunLevel.SCENARIO is not (will not be) compatible with the Jira Xray json importer.

When using CourgetteRunLevel.FEATURE it's straightforward for Courgette to combine all parallel reports at the end of the test run so the final Cucumber json report will be accepted by the Jira Xray importer as it's in the original Cucumber format.

When using CourgetteRunLevel.SCENARIO, it's a bit different in that Courgette runs each scenario as a separate test so when combining reports at the end of the test run, each report will be included as a new (separate) test in the Cucumber json report. It's not straightforward to associate each scenario to its corresponding feature at the end of the test run.

Have you tried using the Cucumber JUnit plugin and then importing the Cucumber XML into Jira Xray instead?

https://docs.getxray.app/display/XRAY/Import+Execution+Results#ImportExecutionResults-JUnitXMLoutputformat

@RunWith(Courgette.class)
@CourgetteOptions(
        runLevel = CourgetteRunLevel.SCENARIO,
        cucumberOptions = @CucumberOptions(
                ...
                plugin = {
                   "junit:build/cucumber-report/cucumber.xml"
                }
        ))
public class CourgetteRunner {
}

[prashant-ramcharan]

Closing inactive issue.