Parallel execution using maven-failsafe-plugin

FrancisJoshuaCervantes commented 4 years ago

Good day @wakaleo,

I would like to ask if how can I configure parallel execution and configuration on this serenity-cucumber5? Because I am using maven-failsafe version 3.0.0-M4. I also tried using profiles but then I am not successful to execute in parallel. Hoping for your response. Thank you.

azmo-rinsler commented 4 years ago

Hello @wakaleo , I think I might be experiencing this same (or a very similar) difficulty:

I'm trying to update to the new version for the Cucumber 5 support, and getting failures any time the -Dparallel.tests arg is set to anything greater than 1. No tests get run, and I get one of these errors for each thread that tries to run:

java.lang.NullPointerException: null
    at java.base/java.util.Objects.requireNonNull(Objects.java:221) ~[na:na]
    at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178) ~[na:na]
    at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[na:na]
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) ~[na:na]
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[na:na]
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[na:na]
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[na:na]
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[na:na]
    at net.serenitybdd.cucumber.suiteslicing.CucumberScenarioLoader.load(CucumberScenarioLoader.java:42) ~[serenity-cucumber5-2.2.0.jar:2.2.0]
    at net.serenitybdd.cucumber.suiteslicing.CucumberSuiteSlicer.scenarios(CucumberSuiteSlicer.java:22) ~[serenity-cucumber5-2.2.0.jar:2.2.0]
    at io.cucumber.junit.CucumberSerenityRunner.getChildren(CucumberSerenityRunner.java:285) ~[serenity-cucumber5-2.2.0.jar:2.2.0]
    at org.junit.runners.ParentRunner.getFilteredChildren(ParentRunner.java:426) ~[junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.getDescription(ParentRunner.java:351) ~[junit-4.12.jar:4.12]
    at org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC.isThreadSafe(ParallelComputerBuilder.java:667) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC.getRunner(ParallelComputerBuilder.java:382) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.junit.runner.Computer$1.runnerForClass(Computer.java:31) ~[junit-4.12.jar:4.12]
    at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) ~[junit-4.12.jar:4.12]
    at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:101) ~[junit-4.12.jar:4.12]
    at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:87) ~[junit-4.12.jar:4.12]
    at org.junit.runners.Suite.<init>(Suite.java:81) ~[junit-4.12.jar:4.12]
    at org.junit.runner.Computer.getSuite(Computer.java:28) ~[junit-4.12.jar:4.12]
    at org.apache.maven.surefire.junitcore.pc.ParallelComputerBuilder$PC.getSuite(ParallelComputerBuilder.java:353) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.junit.runner.Request.classes(Request.java:75) ~[junit-4.12.jar:4.12]
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:126) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) ~[surefire-junit47-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377) ~[surefire-booter-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138) ~[surefire-booter-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465) ~[surefire-booter-3.0.0-M4.jar:3.0.0-M4]
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451) ~[surefire-booter-3.0.0-M4.jar:3.0.0-M4]

I've tried going through my pom file and cleaning things up a bit, but there are probably a few comments hanging out from where I was trying to find a combination of settings that worked:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.idexx</groupId>
    <artifactId>qe-lynxx-automation-suite</artifactId>
    <version>2.0.0</version>
    <packaging>jar</packaging>

    <name>QE LYNXX Automation Suite</name>

    <properties>
        <!-- Dependencies related to the application model -->
        <application.model.version>4.23.12</application.model.version>

        <!-- Number of test runners to execute in parallel -->
        <!-- ( Should generally be set at runtime using the -Dparallel.tests={# of available UFT licenses} arg -->
        <parallel.tests>1</parallel.tests>

        <!-- Dependencies related to batching -->
        <maven.surefire.version>3.0.0-M4</maven.surefire.version>
        <maven.failsafe.version>3.0.0-M4</maven.failsafe.version>
        <!--maven.surefire.version>2.22.2</maven.surefire.version>
        <maven.failsafe.version>2.22.2</maven.failsafe.version-->

        <!-- Dependencies related to Serenity and Cucumber -->
        <serenity.core.version>2.2.5</serenity.core.version>
        <serenity.maven.version>2.2.5</serenity.maven.version>
        <serenity.cucumber.version>2.2.0</serenity.cucumber.version>

        <logback.classic.version>1.0.13</logback.classic.version>
        <junit.version>4.12</junit.version>
        <assertj.version>3.6.2</assertj.version>
        <hamcrest.version>1.3</hamcrest.version>

        <maven.compiler.version>3.8.1</maven.compiler.version>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>

        <!-- Additional Dependencies -->
        <json.version>1.1.1</json.version>
        <faker.version>1.0.1</faker.version>
        <aws.bom.version>2.5.29</aws.bom.version>
        <awaitility.version>4.0.2</awaitility.version>
        <log4j.apache.version>2.12.1</log4j.apache.version>
        <rest.assured.version>3.1.1</rest.assured.version>
        <oracle.jdbc.version>11.2.0.3</oracle.jdbc.version>
        <jcifs.version>2.1.4</jcifs.version>
        <jakarta.version>2.3.3</jakarta.version>
        <jaxws.version>2.3.2</jaxws.version>

        <!-- Other Stuff -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <encoding>UTF-8</encoding>
        <webdriver.base.url/>
        <tags/>
    </properties>

    <!-- AWS SDK BOM used by individual AWS dependencies later on -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.bom.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <!-- Dependencies related to the application model -->
        <dependency>
            <groupId>com.idexx</groupId>
            <artifactId>qe-lynxx-application-model</artifactId>
            <version>${application.model.version}</version>
        </dependency>

        <!-- Dependency used by LynxxTestLogger Utility Class -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j.apache.version}</version>
        </dependency>

        <!-- Dependencies related to CucumberWithSerenity -->
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>${logback.classic.version}</version>
        </dependency>

        <dependency>
            <groupId>net.serenity-bdd</groupId>
            <artifactId>serenity-core</artifactId>
            <version>${serenity.core.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>net.serenity-bdd</groupId>
            <artifactId>serenity-junit</artifactId>
            <version>${serenity.core.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>net.serenity-bdd</groupId>
            <artifactId>serenity-screenplay</artifactId>
            <version>${serenity.core.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>net.serenity-bdd</groupId>
            <artifactId>serenity-screenplay-webdriver</artifactId>
            <version>${serenity.core.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>net.serenity-bdd</groupId>
            <artifactId>serenity-cucumber5</artifactId>
            <version>${serenity.cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.assertj</groupId>
            <artifactId>assertj-core</artifactId>
            <version>${assertj.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-all</artifactId>
            <version>${hamcrest.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Additional Dependencies (Utility Classes, etc.) -->
        <dependency>
            <groupId>org.awaitility</groupId>
            <artifactId>awaitility</artifactId>
            <version>${awaitility.version}</version>
        </dependency>

        <dependency>
            <groupId>com.googlecode.json-simple</groupId>
            <artifactId>json-simple</artifactId>
            <version>${json.version}</version>
        </dependency>

        <dependency>
            <groupId>com.github.javafaker</groupId>
            <artifactId>javafaker</artifactId>
            <version>${faker.version}</version>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>secretsmanager</artifactId>
        </dependency>

        <dependency>
            <groupId>eu.agno3.jcifs</groupId>
            <artifactId>jcifs-ng</artifactId>
            <version>${jcifs.version}</version>
        </dependency>

        <dependency>
            <groupId>io.rest-assured</groupId>
            <artifactId>rest-assured</artifactId>
            <version>${rest.assured.version}</version>
        </dependency>

        <dependency>
            <groupId>com.oracle</groupId>
            <artifactId>ojdbc6</artifactId>
            <version>${oracle.jdbc.version}</version>
        </dependency>

        <dependency>
            <groupId>jakarta.xml.ws</groupId>
            <artifactId>jakarta.xml.ws-api</artifactId>
            <version>${jakarta.version}</version>
        </dependency>

        <dependency>
            <groupId>com.sun.xml.ws</groupId>
            <artifactId>jaxws-ri</artifactId>
            <version>${jaxws.version}</version>
            <type>zip</type>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.version}</version>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>

            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>${maven.failsafe.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>

                        <configuration>
                            <includes>
                                <include>**/SlicedTestSuite*.java</include>
                            </includes>
                            <systemPropertyVariables>
                                <webdriver.base.url>${webdriver.base.url}</webdriver.base.url>

                                <!-- not sure about all these - still fiddling around w/ them...  -->
                                <serenity.test.statistics.dir>/aggregated</serenity.test.statistics.dir>

                                <!-- Batching generates multiple JVM processes, each of which goes to a different VM -->
                                <serenity.batch.count>0${parallel.tests}</serenity.batch.count>
                                <!-- IntelliJ is claiming surefire.forkNumber doesn't exist, but this does not appear
                                     to be the case, considering batching appears to be working just fine...?        -->
                                <!--suppress UnresolvedMavenProperty -->
                                <serenity.batch.number>0${surefire.forkNumber}</serenity.batch.number>

                                <!-- How would forking work w/ Jenkins distributing commands to UFT on multiple VMs? -->
                                <!--serenity.fork.count>0${parallel.tests}</serenity.fork.count-->
                                <!--serenity.fork.number>0${surefire.forkNumber}</serenity.fork.number-->
                                <!-- end of unsure -->
                            </systemPropertyVariables>

                            <threadCount>${parallel.tests}</threadCount>
                            <forkCount>${parallel.tests}</forkCount>
                            <parallel>classes</parallel>

                        </configuration>

                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.version}</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>net.serenity-bdd.maven.plugins</groupId>
                <artifactId>serenity-maven-plugin</artifactId>
                <version>${serenity.maven.version}</version>
                <configuration>
                    <tags>${tags}</tags>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>net.serenity-bdd</groupId>
                        <artifactId>serenity-core</artifactId>
                        <version>${serenity.core.version}</version>
                    </dependency>
                </dependencies>
                <executions>
                    <execution>
                        <id>serenity-reports</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>aggregate</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Is there any other information I can provide to help?

wakaleo commented 4 years ago

Are you trying to run batches of tests across multiple machines?

azmo-rinsler commented 4 years ago

Yeah, the application I have to test is unfortunately a desktop client, and we can only run 1 test at a time per machine.

Our solution was to set up some VM's and batch the execution out to them using Jenkins, and programmatically generate the appropriate number of SlicedTestRunner classes as needed to match the value of -Dparallel.tests passed in at runtime.

wakaleo commented 4 years ago

Hmmm, I'm not familiar with that particular bit of code: maybe @nickbarrett or @cliviu could shed some light?

NickBarrett commented 4 years ago

Sorry folks. I would love to help but I haven't done any Java since university. I suspect my basic late-1990's Java skills won't help here. 😅 All the best finding a solution 👍

wakaleo commented 4 years ago

Sorry @NickBarrett , wrong Nick, I meant @nbarrett!

nbarrett commented 4 years ago

Hi @NickBarrett - sorry you got lumbered with the responsibility of fixing this problem, I'm taking a look now! 😆

NickBarrett commented 4 years ago

Phew that's a huge relief, for everybody 🤣

azmo-rinsler commented 4 years ago

This is fantastic, and after updating to 2.2.5 I was able to start running again, thank you!

I'm not sure if this is related (was also happening before I updated to the new version), but there is one other slightly strange behavior with failsafe parallelization; for some reason it's failing to find Example-level tags within Scenario Outlines, but only when using multiple threads.

e.g. here -Dcucumber.filter.tags="@Debug and not @Region:AU" -Dparallel.tests=1 works as expected, but -Dcucumber.filter.tags="@Region:AU or @Region:CA" -Dparallel.tests=2 fails to find anything.

  @Suite:MVP @ALM_ID:4718 @ALM_ID:4723 @Debug
  Scenario Outline: Logging in using the correct user credentials

    Given I have opened Lynxx and waited for the login screen to appear

    When  I enter my user credentials
    Then  The lab selection screen should appear

    When  I select the default lab for the "<region>" region
    Then  The landing screen should appear

   @Region:AU
    Examples:
      | region  |
      | AU      |

   @Region:CA
    Examples:
      | region  |
      | CA      |

It seems like "expected" behavior in that it isn't producing any errors and even manages to produce an (empty) report as output - it just happens that nothing gets included because all of the @Region:... tags are located at the Example-level.

azmo-rinsler commented 4 years ago

Hi @nbarrett - does this seem related, or would it make more sense in / as a different Issue?

I just tried with serenity core and maven version 2.2.7, and serenity cucumber version at 2.2.5

nbarrett commented 4 years ago

Hello again @azmo-rinsler - sorry I've personally had no experience at all with Cucumber 5, but will try and checkout the latest serenity versions this weekend and take a look. My only observation at this point is that (at least when I last used it) Serenity batching and parallel operation only worked using the system parameters as documented here e.g.

serenity.batch.count and serenity.batch.number
and within a machine: serenity.fork.count and serenity.fork.number.

I can see that you've also supplied tags by means of system properties and I'm not sure at what point the tags when applied in this way, would be applied to the filtering of scenarios. As an experiment, what would happen if the tags were instead applied as annotations on the test runner(s) you are using?

Hope this helps somewhat?

ccharnkij commented 4 years ago

@azmo-rinsler I might be able to help if you can provide me with a repeatable code that i can run and reproduce this weird behavior.

azmo-rinsler commented 4 years ago

I tried grabbing the cucumber starter repo and just make a few small changes to keep everything as close to default as possible; put it here for easy access

Trying to run mvn clean verify -Dcucumber.filter.tags="@Category:B" -Dparallel.tests=1 should work as expected, but just mvn clean verify -Dcucumber.filter.tags="@Category:B" uses the default value of <parallel.tests>2</parallel.tests>, and results in no tests being found and run.

The most notable difference is to the pom file; to the maven-failsafe-plugin here:

            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>3.0.0-M4</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <includes>
                                <include>**/*CucumberTestSuite.java</include>
                            </includes>
                            <systemPropertyVariables>
                                <webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
                                <serenity.batch.count>0${parallel.tests}</serenity.batch.count>
                                <serenity.batch.number>0${surefire.forkNumber}</serenity.batch.number>
                            </systemPropertyVariables>
                            <threadCount>${parallel.tests}</threadCount>
                            <forkCount>${parallel.tests}</forkCount>
                            <reuseForks>false</reuseForks>
                            <parallel>methods</parallel>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

ccharnkij commented 4 years ago

@azmo-rinsler I didn't look deep into it, but all I changed were

parallel=classes, instead of method
verify was run with batch information, verify -Dserenity.batch.count=1 -Dserenity.batch.number=1 -Dcucumber.filter.tags=@Category:B

And the test was found.

azmo-rinsler commented 4 years ago

I'm not extremely familiar with Maven, so I might be misunderstanding, but isn't that what this part here in the POM file is doing?

<systemPropertyVariables>
     <webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
     <serenity.batch.count>0${parallel.tests}</serenity.batch.count>
     <serenity.batch.number>0${surefire.forkNumber}</serenity.batch.number>
</systemPropertyVariables>

This also brings up something I hadn't thought about before - it's possible this particular issue might be related to setting serenity.batch.count to greater than 1, and not necessarily directly related to the value of parallel.tests otherwise.

I'm not sure if / how it's managing the batching behind the scenes with this set of runtime args, but at the very least it appears to be correctly finding the tags, so maybe it is the value of serenity.batch.count that matters here?

Finds tests: mvn clean verify -Dcucumber.filter.tags=@Category:B -Dparallel.tests=2 -Dserenity.batch.count=1 -Dserenity.batch.number=1

Does NOT find tests: mvn clean verify -Dcucumber.filter.tags=@Category:B -Dparallel.tests=2 -Dserenity.batch.count=2 -Dserenity.batch.number=1

@nbarrett Sorry - I somehow managed to miss the second part of your last comment. Here's what the test runners look like right now - how should I apply the additional annotation?

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
        plugin = {"pretty"},
        features = "src/test/resources/features",
        strict = true
)
public class AnotherCucumberTestSuite {}

Unless you meant within the Feature files? Tags seem to work reliably at the Feature and Scenario / Scenario Outline levels, while Examples do work, but it seems to only be when serenity.batch.count=1

@ThisWorks
Feature: Search by keyword

  @ThisWorksToo
  Scenario Outline: Searching for a category
    Given Sergey is researching things on the "<category>"
    When  he looks up "Cucumber"
    Then  he should see information about "Cucumber"

    @Category:A
    Examples:
      | category |
      | A1       |
      | A2       |

    @Category:B
    Examples:
      | category |
      | B1       |
      | B2       |

  @ThisWorksAsWell
  Scenario Outline: Creating a new category
    Given Sergey is creating things about the "<category>"
    When  he looks up "Cucumber"
    Then  he should see information about "Cucumber"

    @Category:B
    Examples:
      | category |
      | A1       |
      | A2       |

    @Category:C
    Examples:
      | category |
      | B1       |
      | B2       |

Jazzyekim commented 4 years ago

Hi, guys! Do you plan the release any time soon? We really need that fix for parallel execution and tags

wakaleo commented 4 years ago

Probably later this week.

serenity-bdd / serenity-cucumber5

Parallel execution using maven-failsafe-plugin #5