Features are being generated as "Unknown test suite"

[jorgeejgonzalez]

I'm currently using the allure-cucumber-jvm-adaptor version 1.5.1 to run the listener through a set of Cucumber tests.

When the reports are generated, I notice that only a few scenarios are recognized as behaviours, normally it's the scenarios of 2 random features (out of 10 or 12).

Looking around the allure-results i noticed that only a couple of the testsuite.xml files get the proper format

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:test-suite start="1454325877685" stop="1454325877691" version="1.4.15" xmlns:ns2="urn:model.allure.qatools.yandex.ru">
    <name>Extracting data</name>
    <title>Extracting data</title>
    <test-cases>

But most of the results just show the following format

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:test-suite start="0" stop="1454325853500" version="1.4.15" xmlns:ns2="urn:model.allure.qatools.yandex.ru">
    <test-cases>

Is this a known issue or am I getting something wrong on my gherkin files that are preventing the proper results generation?

I noticed #21 seems to be somewhat related to what I'm getting, but I'm not really sure if we are talking about the same thing in both issues.

[clicman]

@jorgeejgonzalez it is not enough data to analyze. Could you provide more information? I need scenario name for blank test result and exceptions if it present.

[baev]

@jorgeejgonzalez it would be nice if you'll provide us the sample project to reproduce the problem

[dhaerinck]

I encountered the exact same issue and I fixed it by removing all non-ASCII characters from the step definitions. I don't know if that'll work for you, but it's worth a try.

[jorgeejgonzalez]

Ok, to provide a little more perspective, I noticed that it's only the scenarios of two features that get proper naming, if I add a third feature to the mix, the third feature wont get properly named.

But let's address this step by step.

First my pom.xml looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.company</groupId>
    <artifactId>web-testing-framework</artifactId>
    <version>1.2.2</version>
    <packaging>jar</packaging>
    <name>Web Testing Framework</name>

    <properties>
        <apache.poi.version>3.13</apache.poi.version>
        <cucumber.version>1.2.4</cucumber.version>
        <picocontainer.version>2.15</picocontainer.version>
        <allure.version>1.4.15</allure.version>
        <allure.adaptor.version>1.5.1</allure.adaptor.version>
        <aspectj.version>1.8.7</aspectj.version>
        <phantomdriver.version>1.2.1</phantomdriver.version>
        <selenium.version>2.49.0</selenium.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>com.codeborne</groupId>
            <artifactId>phantomjsdriver</artifactId>
            <version>${phantomdriver.version}</version>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-remote-driver</artifactId>
            <version>${selenium.version}</version>
        </dependency>

        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>${selenium.version}</version>
        </dependency>
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-server</artifactId>
            <version>${selenium.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.poi</groupId>
            <artifactId>poi</artifactId>
            <version>${apache.poi.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.poi</groupId>
            <artifactId>poi-ooxml</artifactId>
            <version>${apache.poi.version}</version>
        </dependency>

        <dependency>
            <groupId>ru.yandex.qatools.allure</groupId>
            <artifactId>allure-cucumber-jvm-adaptor</artifactId>
            <version>${allure.adaptor.version}</version>
        </dependency>

        <dependency>
            <groupId>org.easytesting</groupId>
            <artifactId>fest-assert-core</artifactId>
            <version>2.0M10</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>net.masterthought</groupId>
            <artifactId>cucumber-reporting</artifactId>
            <version>1.2.0</version>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.19</version>
                    <dependencies>
                        <dependency>
                            <groupId>org.apache.maven.surefire</groupId>
                            <artifactId>surefire-junit47</artifactId>
                            <version>2.19</version>
                        </dependency>
                        <dependency>
                            <groupId>org.aspectj</groupId>
                            <artifactId>aspectjweaver</artifactId>
                            <version>${aspectj.version}</version>
                        </dependency>
                    </dependencies>

                    <configuration>
                        <testFailureIgnore>true</testFailureIgnore>
                        <argLine>
                            -javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
                        </argLine>
                        <properties>
                            <property>
                                <name>listener</name>
                                <value>ru.yandex.qatools.allure.cucumberjvm.AllureRunListener</value>
                            </property>
                        </properties>
                    </configuration>
                </plugin>

                <plugin>
                    <groupId>net.masterthought</groupId>
                    <artifactId>maven-cucumber-reporting</artifactId>
                    <version>0.1.0</version>
                    <executions>
                        <execution>
                            <id>execution</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>generate</goal>
                            </goals>
                            <configuration>
                                <projectName>web-testing-framework</projectName>
                                <outputDirectory>${project.build.directory}/site/cucumber-reports</outputDirectory>
                                <cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>
                                <enableFlashCharts>true</enableFlashCharts>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>

                <plugin>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>

                <plugin>
                    <groupId>org.eclipse.jetty</groupId>
                    <artifactId>jetty-maven-plugin</artifactId>
                    <version>9.2.10.v20150310</version>
                    <configuration>
                        <webAppSourceDirectory>${project.build.directory}/site/allure-maven-plugin</webAppSourceDirectory>
                        <stopKey>stop</stopKey>
                        <stopPort>1234</stopPort>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>

    <reporting>
        <excludeDefaults>true</excludeDefaults>
        <plugins>
            <plugin>
                <groupId>ru.yandex.qatools.allure</groupId>
                <artifactId>allure-maven-plugin</artifactId>
                <version>2.0</version>
            </plugin>
        </plugins>
    </reporting>

</project>

My features are:

ExtractBrandPagesData.feature

@setup
Feature: We need to extract the Brand Page Data
  Scenario: Extracting data from famous smoke
    Given I need to Setup the Features
    And I navigate to the page "https://www...."
    And I crawl through the list
    When I store the extracted data
    Then I should process the features

PageBreadcrumbs.feature

@actionvalidation
Feature: In every Page all breadcrumbs must go to correct links

  Scenario Outline: I will validate the navigation of the breadcrumbs
    Given I want to check the breadcrumbs of "<URL>"
    When I check all the breadcrumbs
    Then the navigation must be valid

    Examples:
    | URL |
    | https://www.... |
| https://www... |

PageValidTitleLength.feature

@seo
Feature: The Title length of all Pages should be in a rage

  Scenario Outline: I will validate the length of the Page Title
    Given I want to the check the content of the url "<URL>"
    When I navigate to the page
    Then the title should be  under 70 characters
    Examples:
      | URL |
  | https:... |
| https:.. |

Now, I actually have many more features but just with these three I can replicate the error.

If I test with the three features the title length scenarios do not generate the name and title tags

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:test-suite start="0" stop="1454539423542" version="1.4.15" xmlns:ns2="urn:model.allure.qatools.yandex.ru">
    <test-cases>
        <test-case start="1454539422112" stop="1454539422112" status="passed">
            <name>Given I want to the check the content of the url "https://www....."</name>
            <title>Given I want to the check the content of the url "https://www...."</title>
            <steps/>
            <attachments/>
            <labels>

But If I remove one of the other features, then they get named properly

<ns2:test-suite start="1454539293366" stop="1454539296118" version="1.4.15" xmlns:ns2="urn:model.allure.qatools.yandex.ru">
    <name>I will validate the length of the Page Title | https://www..... |</name>
    <title>I will validate the length of the Page Title | https://www..... |</title>
    <test-cases>
        <test-case start="1454539293376" stop="1454539293378" status="passed">
            <name>Given I want to the check the content of the url "https://www...."</name>
            <title>Given I want to the check the content of the url "https://www....."</title>
            <steps/>
            <attachments/>
            <labels>

I doubt this is a problem of encoding but it seems to me to be some sort of multithreading issue between the cucumber-jvm runner and the listener.

Just so you know, these are the runners I'm using

package com.company.automation;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
        features = {"classpath:features-processed"},
        plugin = {"pretty", "html:target/site/cucumber-pretty","json:target/cucumber.json"}
)
public class FeaturesTest {
}

package com.company.automation;

import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
        features = {"classpath:features/setup"},
        plugin = {"pretty", "html:target/site/cucumber-pretty","json:target/cucumber.json"},
        tags = "@setup"
)
public class FeaturesSetupTest {
}

Regards

NOTE: Edited out some data sensible for my customer

[clicman]

I doubt this is a problem of encoding but it seems to me to be some sort of multithreading issue between the cucumber-jvm runner and the listener.

I don`t see multithreading configuration on pom.xml. Are you sure what tests runs in parallel?

[jorgeejgonzalez]

I didn't enable parallel running but I can't figure out anything else, if it was due to encoding then the error should be persistant, instead only two features get properly named in each run

[Jiwari]

I found some problems when using different files with the same Features names; when the second Feature with the same name of one executed previously was run, the Scenarios under this Feature were classified under a Unknown suite. Per your sample files, it doesn't seem to be your problem, but check your project for repeated Features names, that might be the problem.

[clicman]

I confirm problem. It appears if you have duplicate scenario names - not features. If I place into folder features-setup feature A with scenario B and same feature in folder features-processed ant then start tests - problem reproduced. But if I change scenario name in folder features-setup (and leave feature name) problem resloved.

@Jiwari you can use it as workaround until I realese bugfix.

[jorgeejgonzalez]

I've been checking my scenarios and each one has an unique name and the error still occurs; maybe it has something to do with the fact that I'm using scenario outlines. Not sure if that triggers the issue

[clicman]

Scenario outlines may leads to this error only same example liness will in one outline.

[clicman]

And what difference between classpath:features/setup and classpath:features-processed?

[jorgeejgonzalez]

This is a big difference to my project; our features are dynamically generated from a group of templates after scrapping some data from the web, particularly, the urls to which we are going to perform the validations.

Hence the setup feature which performs the data extraction and creates the features from the templates.

You can also find the repository with the code in my github account under the name of WebTestingFramework

[clicman]

wow! I found the problem! It happens when we have two features with scenario outlines with same parameters. Problem found. I wil try to solve it ASAP.

As workaround (tested) you can generate additional cloumn for dummy parameter with some unique value. For example:

Feature 1:
Outline:
|URL|UUID|
|http://fb.com|UUID=234321421|
Feature 2:
Outline 2:
|URL|UUID|
|http://fb.com|UUID=456523423|

It will affect on scenario name only.

P.S. Thank you for test project.

[clicman]

Quick research leads to conclusion what it cant be fixed on adaptor side. So Ive created an issue in https://github.com/cucumber/cucumber-jvm/issues/959 . So until it accepted, use workaround please.

[jorgeejgonzalez]

I can confirm that the workaround is working.

Thanks

[SButterfly]

I'm also facing this problem. And I'd like to mention, there is no need of different features. It also reproduced in one:

Feature:
Outline 1:
|URL|
|http://fb.com|
Outline 2:
|URL|
|http://fb.com|

Any news about fixing it?

[clicman]

@SButterfly yes I am working on it. You could try this branch. But I won`t merge it into maser because it have issues with multithreading. To make it work you should follow special pom.xml configuration

If you will decide to try this, give me some feedback please - is it solves problem for you or not.

[SButterfly]

@clicman Well, now there are no 'Without feature' row in allure reports and those broken scenario outlines are grouped now. But when I opened logs, I saw

122 Scenarios (9 failed, 113 passed)
690 Steps (8 failed, 13 skipped, 669 passed)

And on xUnit tab at allure reports I saw:

Total: 110 test suites, 690 test cases

Smth gone wrong

[clicman]

So... Which one of provided results are actual and which are wrong? I admit it count wrong cases count in logs but should count it right in allure report.

[SButterfly]

122 is correct. I thought it is allure report bug. Because on 'Behaviors tab' total number of scenarios is 122. But on xUnit tab total number is 110, which is wrong.

[SButterfly]

What about current bug? When can we have a release with it?

[clicman]

I should take some conversation with guys from Surefire team. It has some mess with couple surefire issues: https://issues.apache.org/jira/browse/SUREFIRE-1227 https://issues.apache.org/jira/browse/SUREFIRE-1187 https://issues.apache.org/jira/browse/SUREFIRE-1095

So if I release it now, people who uses parallel execution will start to catch bugs.

[SButterfly]

Hello, It's me.. again.. I found bug in new implementation It looks like if two scenarios have similar names, they placed under one scenario in allure reports.

Ex

Feature: Feature

  Scenario: Scenario
    When whe run the scenario

  Scenario: Scenario
    Then scenario name shuld be complete

Then in allure reports at 'Behaviors' tab.

Feature: Feature
+- Scenario: Scenario
    +- When whe run the scenario
    +- Then scenario name shuld be complete

[clicman]

@SButterfly it is not an implementation bug :) Its how allure sort entities. Also it is strange to have scenarious with same name in one feature. It is breaks BDD principle (and allure and everything). Do not do that anymore.

[clicman]

Fixed in master. Will available in 1.6 release