Xceptance / neodymium-library

A test automation library based on common other best practice open source libraries. It adds missing functionalities but does not reinvent the wheel. Just glues stuff together nicely and adds some sprinkles.
MIT License
80 stars 11 forks source link

# 164 fix error displaying in the allure report #166

Closed oomelianchuk closed 6 months ago

oomelianchuk commented 3 years ago

Here is a solution to fix error displaying on the allure report. This code makes the very first test failure to be displayed as a failure cause. The stack traces of errors from further steps are now attached to the step as an HTML file, which allows user to view them inside the report.

This solution requires to change the listener properties of maven-surefire-plugin from io.qameta.allure.junit4.AllureJunit4 to com.xceptance.neodymium.AllureNeodymium

Here you can view the reports, generated with this code:

Simple test Test with steps Test with steps and selenide error The same test in 2 maven forks

oomelianchuk commented 3 years ago

It's possible to replace StepListener and META-INF manipulations with a customized AllureSelenide listener, but this will only work for Selenide failures (or for assertions wrapped with SelenideAddons.wrappAssertion). On the other hand, we'll get rid of the usage of deprecated code.

IMO, there is no problem to ask the user to replace the listener, if they want the very first failure to be displayed on the top of the overview tab, because we have some articles in our Wiki, which tells, how to configure Allure for the project. We could mention the option to use our listener to improve the reports.

This might be also possible to adjust Allure behavior with a custom plugin. But I suppose that this will also require some configuration effort from the user and it might be more complex than changing the listener.

Anyway, fix of this bug will make Neodymium be more dependant on Allure than it is now because the reason for the bug is the way AllureJunit4 handles the test failures. It simply overwrites the test failure description with every failure, so the last failure appears to be the main failure reason, which is not always the case.

oomelianchuk commented 7 months ago

using the latest version of Allure and Allure report fixes the issue. Please, mind that although the latest version of allure-maven plugin is 2.12.0, it required to use 2.27.0 report version, meaning:

      <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>2.12.0</version>
        <configuration>
          <reportVersion>2.27.0</reportVersion>
          <resultsDirectory>${project.build.directory}/allure-results</resultsDirectory>
        </configuration>
      </plugin>

Therefore, we can close the issue but update of neodymium-example is required

kqmpetenz commented 6 months ago

I added the snippet to the pom of neo example and neo template

wurzelkuchen commented 6 months ago

closes as suggested