allure-framework / allure-java

Allure integrations for Java test frameworks
Apache License 2.0
341 stars 218 forks source link

🐞: [Junit5] Parameterized test failed in @BeforeEach lacks parameters data in result.json #1061

Open ar4development opened 5 days ago

ar4development commented 5 days ago

What happened?

Say we have the code like this:

import io.qameta.allure.Step
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.DisplayName
import org.junit.jupiter.api.Tag
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.CsvSource

class ExperimentTest {

    @BeforeEach
    fun setUp() {
        throw Exception("Like fail in setup")
    }

    @ParameterizedTest(name = "{0} - {1}")
    @DisplayName("Test stats with parameters")
    @CsvSource(value = [
        "val1-1, val2-1",
    ])
    @Tag("testops_test")
    fun test1(val1: String, val2: String) {
        doTestStepFor("$val1 - $val2")
    }

    @Step("Doing test step for {stepSuffix}")
    private fun doTestStepFor(stepSuffix: String) {
        println(">>>>>>> Test step done: $stepSuffix")
    }

}

Run the test and observe result.json.

OBSERVED: There is no parameter data in parameters like:

  "parameters": [
    {
      "name": "UniqueId",
      "value": "[engine:junit-jupiter]/[class:testops_test.ExperimentTest]/[test-template:test1(java.lang.String, java.lang.String)]/[test-template-invocation:#1]",
      "mode": "hidden"
    }
  ],

This is kind of a problem since when the test passes after retry (say we have commented the exception line and run test once again), the parameters are there for successful result.json

  "parameters": [
    {
      "name": "UniqueId",
      "value": "[engine:junit-jupiter]/[class:testops_test.ExperimentTest]/[test-template:test1(java.lang.String, java.lang.String)]/[test-template-invocation:#1]",
      "mode": "hidden"
    },
    {
      "name": "val1",
      "value": "val1-1"
    },
    {
      "name": "val2",
      "value": "val2-1"
    }
  ],

and TestOps fails to merge those two test runs into a single retry chain.

What Allure Integration are you using?

allure-junit5

What version of Allure Integration you are using?

2.25.0

What version of Allure Report you are using?

2.25.0

Code of Conduct