search

trivago / cucable-plugin

Maven plugin that simplifies running Cucumber scenarios in parallel.
Apache License 2.0
115 stars 42 forks source link

Running Incorrect Tag #123

Closed Naumansh closed 5 years ago

Naumansh commented 5 years ago

Running Incorrect Tags All scenarios are executed one by one instead of the specific tag that is supposed to run.

Steps 1- Run scenario @validLogin3 using mvn clean verify -Dtags=@validLogin3 2- All 3 scenarios get executed one by one starting from first one.

Highlighted areas to specifically look in shared consoles logs

Feature File: (Containing test scenarios to test this issue)

@validLogin1 Scenario: test 1 Given I am trying to logg And I am successfull @validLogin2 Scenario: test 2 Given I click "Login/Register" button

Given I login as "category2" user

  #Then I see homepage
**@validLogin3**
Scenario: test 3
 Given I am trying to log
  And I am successful

POM:

org.apache.maven.plugins maven-surefire-plugin true org.apache.maven.plugins maven-failsafe-plugin ${maven.failsafe.plugin.version} Run parallel tests integration-test integration-test verify ${threads} true true -Dfile.encoding=UTF-8 com.trivago.rta cucable-plugin ${cucable.version} generate-test-resources generate-test-resources parallel src/test/java/sba/sbir/steps/CustomCreator.java src/test/resources/features ${generated.feature.directory} ${generated.runner.directory} ${tags} not @wip scenarios off org.codehaus.mojo build-helper-maven-plugin ${maven.build.helper.plugin.version} add-test-source generate-test-sources add-test-source ${generated.runner.directory}

Console Logs

** Framework Logs **

[Info:ETCAF] Scenario Number in Current Execution: 1 [Info:ETCAF] Scenario Name: [@login, @validLogin2] [Info:ETCAF]Controlling DI instance # 635288507 [Info:ETCAF]Controlling WebDriver instance # 355885103 Given I click "Login/Register" button # EyeThinkSteps.handleClick(String) java.lang.NullPointerException at eyethink.automation.bots.analyzer.EyeThink.(EyeThink.java:29) at eyethink.automation.bots.controller.CoreHandlers.eyeThink(CoreHandlers.java:54) at eyethink.automation.bots.bdd.EyeThinkSteps.handleClick(EyeThinkSteps.java:52) at ✽.I click "Login/Register" button(file:target/parallel/features/login_scenario002_run001_IT.feature:8)

Failed scenarios: target/parallel/features/login_scenario002_run001_IT.feature:7 # test 2

1 Scenarios (1 failed) 1 Steps (1 failed) 0m4.815s

java.lang.NullPointerException at eyethink.automation.bots.analyzer.EyeThink.(EyeThink.java:29) at eyethink.automation.bots.controller.CoreHandlers.eyeThink(CoreHandlers.java:54) at eyethink.automation.bots.bdd.EyeThinkSteps.handleClick(EyeThinkSteps.java:52) at ✽.I click "Login/Register" button(file:target/parallel/features/login_scenario002_run001_IT.feature:8)

Jul 09, 2019 9:46:23 AM cucumber.runtime.java.ObjectFactoryLoader loadSelectedObjectFactory WARNING: Use deprecated reflections to load requested ObjectFactory. @login Feature: Login Based Test Cases

@login @validLogin3 Scenario: test 3 # target/parallel/features/login_scenario003_run001_IT.feature:7 Jul 09, 2019 9:46:23 AM org.openqa.selenium.remote.DesiredCapabilities chrome INFO: Using new ChromeOptions() is preferred to DesiredCapabilities.chrome() Starting ChromeDriver 75.0.3770.8 (681f24ea911fe754973dda2fdc6d2a2e159dd300-refs/branch-heads/3770@{#40}) on port 43570 Only local connections are allowed. Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code. Jul 09, 2019 9:46:24 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: W3C

** Framework Logs **

[Info:ETCAF] Scenario Number in Current Execution: 2 [Info:ETCAF] Scenario Name: [@login, @validLogin3] [Info:ETCAF]Controlling DI instance # 827035437 [Info:ETCAF]Controlling WebDriver instance # 1716164115 Step 1 Given I am trying to log # LoginSteps.test1() And I am successful # LoginSteps.test2() java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.fail(Assert.java:96) at sba.sbir.steps.LoginSteps.test2(LoginSteps.java:36) at ✽.I am successful(file:target/parallel/features/login_scenario003_run001_IT.feature:9)

Failed scenarios: target/parallel/features/login_scenario003_run001_IT.feature:7 # test 3

1 Scenarios (1 failed) 2 Steps (1 failed, 1 passed) 0m3.157s

java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.fail(Assert.java:96) at sba.sbir.steps.LoginSteps.test2(LoginSteps.java:36) at ✽.I am successful(file:target/parallel/features/login_scenario003_run001_IT.feature:9)

Jul 09, 2019 9:46:26 AM cucumber.runtime.java.ObjectFactoryLoader loadSelectedObjectFactory WARNING: Use deprecated reflections to load requested ObjectFactory. @login Feature: Login Based Test Cases

@login @validLogin1 Scenario: test 1 # target/parallel/features/login_scenario001_run001_IT.feature:7 Jul 09, 2019 9:46:26 AM org.openqa.selenium.remote.DesiredCapabilities chrome INFO: Using new ChromeOptions() is preferred to DesiredCapabilities.chrome() Starting ChromeDriver 75.0.3770.8 (681f24ea911fe754973dda2fdc6d2a2e159dd300-refs/branch-heads/3770@{#40}) on port 45201 Only local connections are allowed. Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code. Jul 09, 2019 9:46:27 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: W3C

** Framework Logs **

[Info:ETCAF] Scenario Number in Current Execution: 3 [Info:ETCAF] Scenario Name: [@login, @validLogin1] [Info:ETCAF]Controlling DI instance # 2031927175 [Info:ETCAF]Controlling WebDriver instance # 176041373 Step 1_1 Given I am trying to logg # LoginSteps.test1_1() And I am successfull # LoginSteps.test1_2() java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.fail(Assert.java:96) at sba.sbir.steps.LoginSteps.test1_2(LoginSteps.java:47) at ✽.I am successfull(file:target/parallel/features/login_scenario001_run001_IT.feature:9)

Failed scenarios: target/parallel/features/login_scenario001_run001_IT.feature:7 # test 1

1 Scenarios (1 failed) 2 Steps (1 failed, 1 passed) 0m2.302s

java.lang.AssertionError at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.fail(Assert.java:96) at sba.sbir.steps.LoginSteps.test1_2(LoginSteps.java:47) at ✽.I am successfull(file:target/parallel/features/login_scenario001_run001_IT.feature:9)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.606 s - in TestSuite [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

Naumansh commented 5 years ago

Sorry about the bad formatting... I have no clue why it's getting changed to this bad formatting after saving.

bischoffdev commented 5 years ago

Please use github's code formatting with three back ticks (`) before and after your code:

this is an example

tags is not a Cucable option. To run specific tags with Cucable, you need to use the option includeScenarioTags as explained in the documentation: https://github.com/trivago/cucable-plugin#includescenariotags.

Also it is a good idea to use the example project first in order to understand Cucable's usage: https://github.com/trivago/cucable-plugin/tree/master/example-project

Naumansh commented 5 years ago

Are you suggesting me to using scenarios names inside my pom ? Thats not what we do.

I am specifying <includeScenarioTags>${tags}</includeScenarioTags>

Where tags is only something i am passing from my maven command with -Dtags=@myScenario. How is it wrong ?

Naumansh commented 5 years ago 

                <groupId>com.trivago.rta</groupId>
                <artifactId>cucable-plugin</artifactId>
                <version>${cucable.version}</version>
                <executions>
                    <execution>
                        <id>generate-test-resources</id>
                        <phase>generate-test-resources</phase>
                        <goals>
                            <goal>parallel</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <sourceRunnerTemplateFile>src/test/java/sba/sbir/steps/CustomCreator.java</sourceRunnerTemplateFile>
                    <sourceFeatures>src/test/resources/features</sourceFeatures>
                    <generatedFeatureDirectory>${generated.feature.directory}</generatedFeatureDirectory>
                    <generatedRunnerDirectory>${generated.runner.directory}</generatedRunnerDirectory>
                    <includeScenarioTags>${tags}</includeScenarioTags>
                    <includeScenarioTags>not @wip</includeScenarioTags>
                    <parallelizationMode>scenarios</parallelizationMode>
                    <logLevel>off</logLevel>
                </configuration>
            </plugin>```
bischoffdev commented 5 years ago

You are overriding your includeScenarioTags in the line directly under it:

<includeScenarioTags>${tags}</includeScenarioTags>
<includeScenarioTags>not @wip</includeScenarioTags>

So this is ALWAYS running

<includeScenarioTags>not @wip</includeScenarioTags>
Naumansh commented 5 years ago

So my understanding from your description is that if there is a tag that i want to skip during smoke or regression execution i need to specify it as <includeScenarioTags>not @wip</includeScenarioTags> This is really confusing parameter. I got it now though.

How to add a scenario tag to be skipped I have tried with with excludeScenarioTags but i see its not a configuration parameter.

And yes once i removed <includeScenarioTags>not @wip</includeScenarioTags>. it solved the issue. but how to add a tag to be skipped

bischoffdev commented 5 years ago

Everything about inclusion and exclusion of tags is answered in the documentation, complete with examples: https://github.com/trivago/cucable-plugin#includescenariotags

Naumansh commented 5 years ago

Agreed. And thats why i raised this question.

The documentation states as below parallelizationMode By default, Cucable uses the parallelizationMode = scenarios meaning that feature files are split into individual scenarios that each have a dedicated runner.

Sometimes it may be desirable, to parallelize complete features. When setting the parallelizationMode = features, only complete features containing all of their source scenarios are generated so each runner runs a complete feature.

<parallelizationMode>features</parallelizationMode> Note: For this mode to work, must specify a directory. Also, includeScenarioTags and excludeScenarioTags cannot be used.

So why am i now not able to use excludeScenarioTags

Naumansh commented 5 years ago

From the Documentation


<plugin>
    <groupId>com.trivago.rta</groupId>
    <artifactId>cucable-plugin</artifactId>
    <version>${cucable-plugin.version}</version>
    <executions>
        <execution>
            <id>generate-test-resources</id>
            <phase>generate-test-resources</phase>
            <goals>
                <goal>parallel</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <!-- Required properties -->
        <sourceRunnerTemplateFile>src/test/resources/parallel/cucable.template</sourceRunnerTemplateFile>
        <sourceFeatures>src/test/resources/features</sourceFeatures>
        <generatedFeatureDirectory>src/test/resources/parallel/features</generatedFeatureDirectory>
        <generatedRunnerDirectory>src/test/java/parallel/runners</generatedRunnerDirectory>

        <!-- Optional properties -->
        <numberOfTestRuns>1</numberOfTestRuns>
        <includeScenarioTags>@includeMe,@includeMeAsWell</includeScenarioTags>                                
        <excludeScenarioTags>@skip</excludeScenarioTags>
        <logLevel>compact</logLevel>

        <desiredNumberOfRunners>2</desiredNumberOfRunners>                                
        <!-- or <desiredNumberOfFeaturesPerRunner>5</desiredNumberOfRunners> -->
    </configuration>    
</plugin>```
bischoffdev commented 5 years ago

Well, this is a leftover in the documentation from the old version of Cucable. I will adapt it now - thanks for noticing! Cucable uses Cucumber expressions since version 1.5.0 and I removed the unneeded excludeScenarioTags parameter because of this. Cucumber expressions support tag exclusion, inclusion and any combination of those.

Naumansh commented 5 years ago

Make sense. So, i can use -Dtags="@test1 and not @test2" to skip @test2