🐞: TestNg throws an error when trying to run dependent tests with withAllureUpload(){}

What happened?

TestNg throws an error when trying to run dependent tests with withAllureUpload(){}

I know that dependent tests violate idempotency and atomicity of tests, but we have a product with complex logic and sometimes it is necessary to make tests dependent, so we use the TestNg framework, which has parameters like dependsOnMethods or dependsOnGroups.

We have integrated with Allure TestOps, but we also want to use the ability of selective test execution. After studying the TestOps documentation, we tried to dynamically generate testng.xml based on testplan.json, which is created by TestOps during selective test execution, so that all tests in the class where the necessary test is located are launched.

The generator gets the full method names from testplan.json, cuts off the method name, and writes the full class name. According to our idea, the results should be updated for both dependent and independent tests in class.

An example of the generated testng.xml file:


<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite thread-count="1" name="Generated TestNg Suite">

<test thread-count="1" name="Generated Test" group-by-instances="true" verbose="5">

<classes>

<class name="ru.sbt.snuil.somemicroservice.test.SomeMicroserviceTest">

<methods>

<include name=".*"/>

</methods>

</class>

</classes>

</test> <!-- Generated Test -->

</suite> <!-- Generated TestNg Suite -->

But when we run such a testng.xml file, we still get an error:

Method "SomeClassTest.someTest1()[pri:0]" depends on nonexistent method "someTest2"

Some mechanism, possibly filtering, breaks the launch of tests in the order we need.We would like to generate testng.xml ourselves based on testplan.json and have it run smoothly and publish results in TestOps. Maybe it can be disabled or do you have a solution for selective execution of dependent tests for TestNg?

What Allure Integration are you using?

allure-testng

What version of Allure Integration you are using?

2.20.0

What version of Allure Report you are using?

2.20.0

Code of Conduct

[X] I agree to follow this project's Code of Conduct

To disable the Allure tests filter, you must unset the following environment variables: ALLURE_TESTPLAN_PATH and AS_TESTPLAN_PATH before you run the tests.

Also, we can support dependent tests in TestNG:

The filtering is done in https://github.com/allure-framework/allure-java/blob/main/allure-testng/src/main/java/io/qameta/allure/testng/AllureTestNgTestFilter.java class.

We need to analyse all the test methods and include all the methods that selected tests depend on in the run. That should do the trick.

Although, what should we do with groups? If some selected test depends on the group, should we include all the tests in such a group?

I tried using 'unset ALLURE_TESTPLAN_PATH and AS_TESTPLAN_PATH before tests run'.

As far as I know, an environment variable cannot be completely unset.

I tried like:


env.ALLURE_TESTPLAN_PATH=''

env.AS_TESTPLAN_PATH=''

env.ALLURE_TESTPLAN_PATH=null

env.AS_TESTPLAN_PATH=null

sh 'export ALLURE_TESTPLAN_PATH=""; export AS_TESTPLAN_PATH=""'

and different variations with unset, but the result the same. Here is an example of the console log:


[Allure] [320683] Job run [https://testops.example.ru/jobrun/601151] started....

[Allure] [320683] Watching on [allure-results] for session [731274]

[Allure] [320683] Save test plan for [1] tests

[Allure] [320683] Test plan with [1] tests created at [/u01/jenkins/somePath/.allure-ee/testplan.json]

+ wget -q http://someserver/testng-xml-genarator.jar   **-----//here we download our testng generator**

+ java -jar testng-xml-genarator.jar '/somePath/.allure-ee/testplan.json'  **-----//here we generate testng.xml**

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite thread-count="1" name="Generated TestNg Suite">

  <test thread-count="1" name="Generated Test" group-by-instances="true" verbose="5">

    <classes>

      <class name="ru.sbt.somemicroservice.test.SomeTest">

        <methods>

          <include name=".*"/>

        </methods>

      </class>

    </classes>

  </test> <!-- Generated Test -->

</suite> <!-- Generated TestNg Suite -->

+ unset ALLURE_TESTPLAN_PATH   **-----//here was different ways to unset variables**

+ unset AS_TESTPLAN_PATH   **-----//here was different ways to unset variables**

+ java -ea -javaagent:lib/aspectjweaver-1.9.7.jar -Dfile.encoding=UTF-8 -jar lib/somemicroservice-tests.jar -d test-output testng/testng_custom_suite.xml

===== Invoked methods

=====

[09:42:52] [n] r.l.LoggingTestListener   ================================

[09:42:52] [n] r.l.LoggingTestListener   ================================

[09:42:52] [n] r.l.LoggingTestListener   TEST RESULTS (timings, errors):

===============================================

    Generated Test

    Tests run: 0, Failures: 0, Skips: 0

===============================================

org.testng.TestNGException:

Method "SomeTest.SomeTest2()[pri:0, instance:ru.sbt.someMicroservice.test.SomeTest@65be88ae]" depends on nonexistent method "ru.sbt.someMicroservice.test.SomeTest.someTest1"

It would be great if you could implement "We need to analyze all the test methods and include all the methods that the selected tests depend on in the run."

About groups, we use them very rarely, but I agree that if a test is related to a group, then the entire group should be run - otherwise, the test result will be incorrect/failed

allure-framework / allure-java