allure-framework / allure-java

Allure integrations for Java test frameworks
Apache License 2.0
353 stars 223 forks source link

Test results are not presented in the execution order same as TestNG does it. #895

Open OlegKuzovkov opened 1 year ago

OlegKuzovkov commented 1 year ago

Describe the bug Our automated testing heavily depends on @BeforeSuite/@BoforeClass/@BeforeMethod methods. When we analyze test results it is crucial for us to see the execution order of tests in the report to quickly understand the sequence of events that cause a failure to happen. When we execute test cases from IDEA, we see test results for classes/beforeMethods/testMethods ordered by the execution: image

In Allure the same report will look completely different in terms of execution order: image

Even with this simple example this looks bad, but in our case the number of test cases is much bigger that makes our life very complicated when in comes to test result analysis.

To Reproduce Steps to reproduce the behavior:

  1. Use the code snippet below and execute test class with testNG:
    
    package com.ctera.auto.tests;

import lombok.extern.log4j.Log4j2; import org.testng.annotations.*;

import static io.qameta.allure.Allure.step;

@Log4j2 public class TestClass { @BeforeSuite public void beforeSuite() throws Exception { log.info("before suite"); } @BeforeClass public void beforeClass() throws Exception { log.info("before class"); }

@BeforeMethod
public void beforeMethod() throws Exception {
   log.info("before method");
    throw new Exception("Before Method failure");
}

@Test
public void test1() {
    step("Step of the test", () -> {
    });
}
@Test
public void test2() {
    step("Step of the test", () -> {
        log.info("in test");
    });
}
@Test
public void test3() {
    step("Step of the test", () -> {
        log.info("in test");
    });
}

@AfterSuite
public void afterSuite(){
    log.info("before suite");
}
@AfterClass
public void afterClass(){
    log.info("after class");
}
@AfterMethod
public void afterMethod(){
    log.info("after method");
}

}


2. Check results in IDEA
3. Generate Allure Report and check results

**Expected behavior**
Test cases should be organized in execution order, so we could easily detect which @BeforeMethod failed which test by just looking at the test results view, the same as TestNG does.

**Screenshots**
Attached above.

**Additional context**
Add any other context about the problem here.
PavelMal commented 1 year ago

@OlegKuzovkov Good day! Could you provide more details please such as IDEA version / libraries versions I tried to reproduce but failed:

I am using:

1. TestNG - 7.7.0
2. allure-testng - 2.23.0
3. allure-java-commons - 2.23.0
4. IDEA - IntelliJ IDEA 2021.1.2 (Community Edition)
Build #IC-211.7442.40, built on June 1, 2021 (yes, I need to update)
image image
baev commented 1 year ago

Allure TestOps doesn't have a sort by execution order (aka start date).