search

serenity-bdd / serenity-cucumber

Cucumber integration for the Serenity BDD Reporting library
Other
78 stars 74 forks source link

Serenity Test Report is not showing any test data and graphs #223

Open amitasil opened 4 years ago

amitasil commented 4 years ago

Total tests executed in Jenkins build: 116

serenity-maven-plugin:2.1.8:aggregate (serenity-reports)

maven-failsafe-plugin:2.22.2:verify

TestResult
wakaleo commented 4 years ago

We can do nothing with a report like this. Try doing a clean build and checking that you are using the correct versions. How do you know 116 tests are executed? Did you see the Serenity banner in the console output?

amitasil commented 4 years ago

Hi @wakaleo , Yes I checked the test count in my console log. I'll do a cleanup build and see if that fix the issue. However, when I ran the same build with only two tests, I can see the report generated successfully and with data and graphs.

This is my pom.xml:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

4.0.0 
<groupId>DTM</groupId>
<artifactId>DTM</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

<name>GRIS-DTM project with Serenity using Cucumber and WebDriver</name>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <serenity.version>2.1.8</serenity.version>
    <serenity.cucumber.version>1.9.51</serenity.cucumber.version>
    <webdriver.driver>chrome</webdriver.driver>
    <webdriver.remote.url>http://localhost:4444/wd/hub</webdriver.remote.url>
    <encoding>UTF-8</encoding>
    <parallel.tests>4</parallel.tests>
</properties>

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
    <repository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray</name>
        <url>https://jcenter.bintray.com</url>
    </repository>
</repositories>
<pluginRepositories>
    <pluginRepository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray-plugins</name>
        <url>https://jcenter.bintray.com</url>
    </pluginRepository>
</pluginRepositories>

<dependencies>
    <dependency>
        <groupId>com.squareup.okhttp3</groupId>
        <artifactId>okhttp-urlconnection</artifactId>
        <version>4.3.1</version>
    </dependency>
    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-core</artifactId>
        <version>${serenity.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-cucumber</artifactId>
        <version>${serenity.cucumber.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.7.7</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.microsoft.sqlserver</groupId>
        <artifactId>mssql-jdbc</artifactId>
        <version>7.4.1.jre8</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.22.2</version>
            <configuration>
                <skip>true</skip>
            </configuration>
        </plugin>
        <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.22.2</version>
            <configuration>
                <systemPropertyVariables>
                    <webdriver.driver>remote</webdriver.driver>
                    <webdriver.remote.url>${webdriver.remote.url}</webdriver.remote.url>
                    <webdriver.remote.driver>${webdriver.driver}</webdriver.remote.driver>
                </systemPropertyVariables>
                <parallel>classes</parallel>
                <threadCount>${parallel.tests}</threadCount>
                <perCoreThreadCount>true</perCoreThreadCount>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>net.serenity-bdd.maven.plugins</groupId>
            <artifactId>serenity-maven-plugin</artifactId>
            <version>${serenity.version}</version>
            <executions>
                <execution>
                    <id>serenity-reports</id>
                    <phase>post-integration-test</phase>
                    <goals>
                        <goal>aggregate</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>com.github.temyers</groupId>
            <artifactId>cucumber-jvm-parallel-plugin</artifactId>
            <version>5.0.0</version>
            <executions>
                <execution>
                    <id>generateRunners</id>
                    <phase>generate-test-sources</phase>
                    <goals>
                        <goal>generateRunners</goal>
                    </goals>
                    <configuration>
                        <!-- Mandatory -->
                        <!-- List of package names to scan for glue code. -->
                        <glue>
                            <package>DTM.steps</package>
                        </glue>
                        <parallelScheme>FEATURE</parallelScheme>
                        <customVmTemplate>src/test/resources/cucumber-serenity-runner.vm</customVmTemplate>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

amitasil commented 4 years ago

Hi, @wakaleo I tried running a cleanup build and then ran the whole test suite, but again I get the same report with no test data or graphs. Looking at my pom.xml, can you please suggest if I am missing something?

If I run the test suite with a fewer number of tests, then I am able to see the report with data and graph in it. Is it the high number of tests failing the report?

wakaleo commented 4 years ago

I've seen Serenity reports with 5000 tests, so the size isn't an issue. I can't see any issue with the pom; check for errors in the logs and possibly for Gherkin syntax errors in your feature files.

If you want parallel execution I would advise moving to cucumber 4 and serenity-cucumber4 as this has built-in support for parallel tests.

amitasil commented 4 years ago

Hi @wakaleo, thanks for your inputs!

This is what I have tried so far:

Please look into the console errors I am getting in case of a good report compared to a bad report.

Console_GoodReport Console_BadReport
wakaleo commented 4 years ago

I can't see anything here. It could be a project-specific issue - are you sure there are no exceptions in the log files?

amitasil commented 4 years ago

Hi @wakaleo,

No errors related to reports in the console log. Is there any other way to get more logs or to see a different place for the logs?

Console log snapshot: [INFO] --- serenity-maven-plugin:2.1.8:aggregate (serenity-reports) @ DTM --- [INFO] Test results for 116 tests generated in 5.1 secs in directory: file:/C:/Program%20Files%20(x86)/Jenkins/jobs/GRIS/workspace/target/site/serenity/ [INFO] ----------------------------------------- [INFO] SERENITY TESTS : ERROR [INFO] ----------------------------------------- [INFO] | Tests executed | 116 [INFO] | Tests passed | 111 [INFO] | Tests failed | 0 [INFO] | Tests with errors | 4 [INFO] | Tests compromised | 0 [INFO] | Tests pending | 0 [INFO] | Tests ignored/skipped | 1 [INFO] ------------------------ | -------------- [INFO] | Total Duration | 6h 26m 35s [INFO] | Fastest test took | 5s 296ms [INFO] | Slowest test took | 5m 22s [INFO] ----------------------------------------- [INFO] [INFO] SERENITY REPORTS [INFO] - Full Report: file:///C:/Program%20Files%20(x86)/Jenkins/jobs/GRIS/workspace/target/site/serenity/index.html [INFO] [INFO] --- maven-failsafe-plugin:2.22.2:verify (default) @ DTM --- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15:29 min [INFO] Finished at: 2020-03-16T10:07:17+11:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.22.2:verify (default) on project DTM: There are test failures. [ERROR] [ERROR] Please refer to C:\Program Files (x86)\Jenkins\jobs\GRIS\workspace\target\failsafe-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Execute Windows batch command' marked build as failure [htmlpublisher] Archiving HTML reports... [htmlpublisher] Archiving at PROJECT level C:\Program Files (x86)\Jenkins\jobs\GRIS\workspace\target\site\serenity to C:\Program Files (x86)\Jenkins\jobs\GRIS\htmlreports\GRIS_20Test_20Automation_20Report

wakaleo commented 4 years ago

Try running maven with the -eX options. It could be a project or environment-specific issue, in which case I would need to look at the project source and Jenkins machine.

amitasil commented 4 years ago

Hi @wakaleo, I have given you permission to view my repository (https://github.com/amitasil/GRIS_DTM_TEST.git) so that you can look at the project source. The issue is not with Jenkins it seems as even on my local computer I am getting the blank report if I run the whole test suite.

Attaching the logs after running the full test suite by shutting down my Selenium Grid. log.txt

amitasil commented 4 years ago

Hi @wakaleo, Did you get a chance to look into the issue or my project?

romerobu commented 4 years ago

@amitasil hI! I recently faced the same issue. I discovered through the logs that individual html report for each test (debug log) were being generated, however the index.html was missing the pie and the results but when I navigated to requirements (upper tab) I had requirements data related to an old execution for an old tag. I tried to execute then with the old tag and report was OK. My serenity core version was 2.2.0. I did gradle clean and updated serenity core version to 2.2.1 and report was generated fine again. I don't know why but requirements tab keep showing old executions information (that feature doesn't even exist in my project now) but at least I can see last execution results.

Qarj commented 4 years ago

We are facing an issue like this also.

I'm pretty sure the problem for us is that we we have multiple Serenity jobs scheduled throughout the day on Jenkins.

Say we kick off a suite of tests that takes an hour to run against the development environment. Later, a small test against production that runs for 1 minute is kicked off by Jenkins at the scheduled time while the first suite of tests is still running, but perhaps nearly finished.

Since the first step of the execution is mvn clean all the screenshots and other test artifacts get deleted. So when the Serenity report is generated for the hour long run most of the files it needs are missing.

I came across this issue on GitHub since I'm searching to try and find out the correct approach for avoiding these types of problems - it seems we need a step in the Groovy to build to a unique folder for the run rather than always using target/site/serenity?