Results are not recorded for tags when ORing tag expression are used

[gouthamvarma1]

ORing: Tags which are comma separated are ORed.

Results are not recorded for tags when using ORing Tags Used tags={"@SmokeTest,@RegressionTest"} --- Reports are not recorded for this tags expression

It will not always simple like executing a single tag. It can be complicated like executing scenarios which are tagged either as @SmokeTest or @RegressionTest

Feature file Example :

@SmokeTest @test @all Scenario: Test scenario1 Given step1 When step2 Then step3

@RegressionTest @test @all Scenario: Test scenario1 Given step11 When step22 Then step33

However the reports are generated for ANDing for the above example ANDing :Tags which are passed in separate quotes are ANDed

Example tags={"@test","@all"} --- Reports are recorded for this tags expression

Version Used

1.9.0-rc.3 1.9.0-rc.3 1.6.13 1.6.13 org.apache.maven.plugins maven-surefire-plugin 2.20.1 true classes 10 Operation System: Windows 10

[wakaleo]

Where are you defining the tags?

[harishgrb]

I am also facing the same issue. The tags are defined above the Scenario Outline.

Eg:

@ListCustomer @Regression @TestCase6 
            Scenario Outline: Verify LIST Customer - By StartIndex and Resultsperpage is NEGATIVE 
                Given the List customer API  
                And Verify the list customer is displaying all the customers By StartIndex<startIndex> and Resultsperpage<ResultsPerPage> 
                Then the response code is <Response Code> 

                Examples: 
            |startIndex |ResultsPerPage | Response Code | 
            |4          |-588           | 400           |        
            |2          |-854           | 400           |   
            |5          |-66            | 400           |   
            |3          |-2             | 400           |       

[wakaleo]

How are you defining the tags when you run the tests?

[gouthamvarma1]

@wakaleo the tags are defined in serenity runner class currently and executed via maven with command clean verify

Example:

package test;

import cucumber.api.CucumberOptions;
import net.serenitybdd.cucumber.CucumberWithSerenity;
import org.junit.runner.RunWith;

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(plugin = { "pretty" }, features = "src/test/resources/features", tags = { "@regression3,@api3" })
public class CucumberTestSuite {
}

[wakaleo]

The annotation should be:

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
    plugin = { "pretty" }, 
    features = "src/test/resources/features", 
    tags = { "@regression3","@api3" }
)
public class CucumberTestSuite {
}

[gouthamvarma1]

@wakaleo : Please correct me If I am wrong

I have a feature file with the below scenarios @TC1 Scenario: Test1 @TC2 Scenari2: Test2

If I want to run both the scenarios my tags will be like tags = {"@TC1,@TC2"} //then both scenarios are executed but the report is not recorded.( Report Generated with Empty/no data )

But If I use tags
tags = {"@TC1","@TC2"} //none of the scenario's executed

[wakaleo]

Yes, As far as I can tell, tags = {"@TC1","@TC2"} will only run scenarios with both tags, and tags = {"@TC1,@TC2"} (which I have never used) runs scenarios with either tag. You would need to check the Cucumber documentation to be sure. For any issues, the @CucumberOptions annotation is managed by the Cucumber-JVM team, not by the Serenity team, so it would be better to raise an issue with them directly.

[gmkrishnaiah]

Hi @wakaleo , I tried the same scenario as mentioned above. I see that the tests are getting executed and report is generated with no data in it. From this I assume that the issue is with the reporting feature of Serenity by using OR condition.

[wakaleo]

I'm not seeing any issues in execution or reporting with the latest versions of serenity-core and serenity-cucumber, using with @CucumberOptions or the cucumber.options property passed in on the command line. Can you produce a sample project that reproduces the issue?

[gouthamvarma1]

@wakaleo I have attached a sample project to reproduce the issue Upon execution in eclipse with build parameters clean verify I see the below output

[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.235 s - in Feature: Example
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- serenity-maven-plugin:1.9.0-rc.3:aggregate (serenity-reports) @ IssueOR ---
[INFO] current_project.base.dir: C:\Users\gpendyala\eclipse-workspace-FedEx\IssueOR
[INFO] Generating test results for 0 tests
[INFO] 0 requirements loaded after 126 ms
[INFO] 0 related requirements found after 126 ms
[INFO] Generating test outcome reports: false
[INFO] Starting generating reports: 139 ms
[INFO] Configured report threads: 40
[INFO] Test results for 0 tests generated in 582 ms
[INFO] 
[INFO] --- maven-failsafe-plugin:2.20:verify (default) @ IssueOR ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.891 s
[INFO] Finished at: 2018-03-02T17:41:38+05:30
[INFO] Final Memory: 46M/566M
[INFO] ------------------------------------------------------------------------

Serenity_Or_Issue.zip

[cliviu]

hi @gouthamvarma1. Currently serenity supports only the "new style" cucumber tags. Please use tags={"@smoketest or @regressiontest"} instead of tags={"@smoketest,@regressiontest"} . For a complete guide of migrating from the old style tags, please check this page : https://github.com/cucumber/cucumber/tree/master/tag-expressions

[wakaleo]

Closing this as we will only be supporting the new style Cucumber tags.

[handresc1127]

I had the same problem, solved it using "cucumber.options" instead of "tags". Examples:

Ok - Generating test results for n tests mvn -Dcucumber.options="--tags ''" clean install mvn -Dcucumber.options="--tags '@Tag1'" clean install mvn -Dcucumber.options="--tags @Tag1" clean install

Error Generating test results for 0 tests mvn -Dtags=@Tag1 clean install