Courgette Test Runner

[MadhuQAAutomation]

Hi @prashant-ramcharan - I have a scenario in my project, where in the batch related feature files should be executed in a sequential manner and the rest of the feature files in parallel as they don't have any dependency on other. I tried creating 2 stages in jenkins pipeline one for sequential and the other one for parallel, by overriding the test runner values using maven command line. But it didn't work as expected, as both are different runs and the reports of the 2nd run is overrides the report of 1st one.

Can you let me know if this is possible with courgette?

Ex:

stage('test:functional-parallel') { steps { script { if (isUnix()) { sh 'mvn test -Dcourgette.threads=10 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.tags="@parallel-execution"' } else { bat 'mvn test -Dcourgette.threads=10 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.tags="@parallel-execution"' } } } } stage('test:functional-sequential') { steps { script { if (isUnix()) { sh 'mvn test -Dcourgette.threads=1 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.tags="not @parallel-execution"' } else { bat 'mvn test -Dcourgette.threads=1 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.tags="not @parallel-execution"' } } } }

[prashant-ramcharan]

Hello,

You using the same reportTargetDir for both runs so this is why the second run overwrites the first run reports.

Try passing a different reportTargetDir at runtime:

 // *** for the sequential run ***
-Dcourgette.reportTargetDir=build-sequential

[MadhuQAAutomation]

Hi Prashant-Ramcharan - If I do so, there will be 2 reports. But my expectation is to have a single report. Can you let me know whether courgette expects only one .json or accepts list of .json files to generate the report?

On Fri, Jan 13, 2023 at 3:38 PM Prashant Ramcharan @.***> wrote:

Hello,

You using the same reportTargetDir for both runs so this is why the second run overwrites the first run reports.

Try passing a different reportTargetDir at runtime:

// for the sequential run -Dcourgette.reportTargetDir=build-sequential

— Reply to this email directly, view it on GitHub https://github.com/prashant-ramcharan/courgette-jvm/issues/367#issuecomment-1381597016, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWIEGE4T7IPORWTE65IDAGLWSESRRANCNFSM6AAAAAATZPQA2E . You are receiving this because you authored the thread.Message ID: @.***>

[prashant-ramcharan]

This cannot be done.. you cannot have 2 separate runs and one single report.

[MadhuQAAutomation]

I think we cannot consider it as 2 separate runs. Rather, its different stages in the Jenkins pipeline for the test automation suite.

stage('test:functional-parallel') { steps { script { if (isUnix()) { sh 'mvn test -Dcourgette.threads=20 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.plugin="json:reports/cucumber/json/cucumber2.json" @."' } else { bat 'mvn test -Dcourgette.threads=20 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.plugin="json:reports/cucumber/json/cucumber2.json" @."' } } } }

stage('test:functional-sequential') { steps { script { if (isUnix()) { sh 'mvn test -Dcourgette.threads=1 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.plugin="json:reports/cucumber/json/cucumber3.json" -Dcucumber.tags="not @parallel-execution"' } else { bat 'bat test -Dcourgette.threads=1 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcucumber.plugin="json:reports/cucumber/json/cucumber3.json" -Dcucumber.tags="not @parallel-execution"' } } } }

When I do as above, there will be 2 .json files created. Will courgette read these json files to generate the report?

The second question I have is to calculate the Courgette Test Statistics for both the runs combined?

Thanks, Madhu

On Fri, Jan 13, 2023 at 4:42 PM Prashant Ramcharan @.***> wrote:

This cannot be done.. you cannot have 2 separate runs and one single report.

— Reply to this email directly, view it on GitHub https://github.com/prashant-ramcharan/courgette-jvm/issues/367#issuecomment-1381697541, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWIEGE45BWX4Y5E5HET5VCDWSE2DXANCNFSM6AAAAAATZPQA2E . You are receiving this because you authored the thread.Message ID: @.***>

[prashant-ramcharan]

I think we cannot consider it as 2 separate runs.

This is 100% separate runs and cannot be considered anything else.

You have 2 stages in your pipeline. Each stage calls mvn test. Every time you run mvn test your trigger a new test run (i.e. Courgette run)

stage('test:functional-parallel') {
 // will call your Courgette runner 
}

stage('test:functional-sequential') {
 // will call your Courgette runner 
}

1 stage = 1 Courgette run = 1 Result (1 report, 1 stats etc)

When I do as above, there will be 2 .json files created. Will courgette read these json files to generate the report?

Courgette does not combine reports from different runs. It only knows about reports and results from the current run.

The second question I have is to calculate the Courgette Test Statistics for both the runs combined?

As previously mentioned, you cannot as they are separate runs. One run doesn't know the stats or details of the other run.

[MadhuQAAutomation]

Ok, thanks for letting me know.

Is there any possibility of implementing this feature?

Thanks, Madhu

On Fri, Jan 13, 2023 at 7:29 PM Prashant Ramcharan @.***> wrote:

I think we cannot consider it as 2 separate runs.

This is 100% separate runs and cannot be considered anything else.

You have 2 stages in your pipeline. Each stage calls mvn test. Every time you run mvn test your trigger a new test run (i.e. Courgette run)

stage('test:functional-parallel') { // will call your Courgette runner }

stage('test:functional-sequential') { // will call your Courgette runner }

1 stage = 1 Courgette run = 1 Result (1 report, 1 stats etc)

When I do as above, there will be 2 .json files created. Will courgette read these json files to generate the report?

Courgette does not combine reports from different runs. It only knows about reports and results from the current run.

The second question I have is to calculate the Courgette Test Statistics for both the runs combined?

As previously mentioned, you cannot as they are separate runs. One run doesn't know the stats or details of the other run.

— Reply to this email directly, view it on GitHub https://github.com/prashant-ramcharan/courgette-jvm/issues/367#issuecomment-1381898026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWIEGE5ETY3OHNMPH7CPMXTWSFNTFANCNFSM6AAAAAATZPQA2E . You are receiving this because you authored the thread.Message ID: @.***>

[prashant-ramcharan]

No, this is not a feature that can be implemented because there is no way to determine the results from separate Courgette / test runs (without persisting the results somewhere)