search

testng-team / testng-eclipse

Eclipse plug-in for TestNG
https://testng.org
194 stars 164 forks source link

testng-failed.xml does not contain failed methods from all the classes while running the tests in parallel. #473

Open deepakguna opened 4 years ago

deepakguna commented 4 years ago

Problem Statement

testng-failed.xml does not contain failed methods from all the classes while running the tests in parallel.

Any relate message in "Error Log" view

"Windows -> Show View -> Others -> Error Log"

The Dependency Management tool for your project

Operating System

TestNG Version 7.1.0

Eclipse version Version: 2019-06 (4.12.0) Build id: 20190614-1200

TestNG for Eclipse Plugin Version: TestNG 7.0.0.201908240652

Expected behavior testng-failed.xml should contain failed methods from all the classes while running the tests in parallel.

JDK 13.0.1 Testng 7.1.0

When running the tests with Testng there are failures from the methods in both Class A & B, how ever testng-failed.xml contains only either of the class? What may go wrong here

Actual behavior testng-failed.xml is inconsistent while running in parallel. On Many occasion it includes failures from only one test/class while there are failures in other test/class.

Test case sample I have two test classes TestRed and TestRed . Each of the class have 2 @test methods and one of them will fail. testng-failed.xml is inconsistent while running in parallel. On Many occasion it includes failures from only one test/class while there are failures in other test/class.

package test;

import org.testng.Assert; import org.testng.annotations.Test;

public class TestRed {

@Test
public void testAInRed() {
    Assert.assertTrue(false, "will fail");
}
@Test
public void testBInRed() {
    Assert.assertTrue(true, "will pass");
}

} Here is my suite xml

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

Below is the testng-failed.xml :

I was not able to reproduce this issue with the test runner. However i was consistently able to reproduce this issue 2/5 times by running from the suite.xml **It works fine in TestNG v 6.14.3 but not in 7.1.0**