Wrongly skipped dependent test methods when using a @Factory constructor with dataProvider

[vadipp]

Hello!

I'm using the (currently latest) TestNG 6.1.1.

I'm having the same issue as Jeff Weiss posted here: http://jira.opensymphony.com/browse/TESTNG-329?focusedCommentId=33613&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_33613

Here is my sample test class:

import org.apache.log4j.Logger;
import org.testng.Assert;
import org.testng.annotations.*;
import org.testng.annotations.Test;

public class TestTest
{
  private final String param;

  private static int invocation = 0;
  private static Logger logger = Logger.getLogger( TestTest.class );

  @Factory( dataProvider = "prov" )
  public TestTest( String param )
  {
    this.param = param;
  }

  @DataProvider( name = "prov" )
  public static Object[][] dataProvider()
  {
    logger.debug( "Provide data" );
    return new Object[][] {
      { "One" },
      { "Two" },
      { "Three" },
    };
  }

  @BeforeClass
  public void prepare()
  {
    logger.debug( "Prepare " + param );
  }

  @Test
  public void test1()
  {
    logger.debug( "Test1 " + param + " BEGIN" );
    invocation++;
    if( invocation == 2 )
    {
      Assert.fail( "Test1 " + param + " FAIL" );
    }
    logger.debug( "Test1 " + param + " OK" );
  }

  @Test( dependsOnMethods = "test1" )
  public void test2()
  {
    logger.debug( "Test2 " + param );
  }

  @AfterTest
  public void clean()
  {
    logger.debug( "Clean " + param );
  }
}

Here is what I get:

Running TestSuite
2011-07-29 16:20:11,840 DEBUG TestTest - Provide data
2011-07-29 16:20:11,930 DEBUG TestTest - Prepare One
2011-07-29 16:20:11,930 DEBUG TestTest - Prepare Two
2011-07-29 16:20:11,931 DEBUG TestTest - Prepare Three
2011-07-29 16:20:19,767 DEBUG TestTest - Test1 One BEGIN
2011-07-29 16:20:25,205 DEBUG TestTest - Test1 One OK
2011-07-29 16:20:38,789 DEBUG TestTest - Test1 Two BEGIN
2011-07-29 16:20:45,929 DEBUG TestTest - Test1 Three BEGIN
2011-07-29 16:20:45,929 DEBUG TestTest - Test1 Three OK
2011-07-29 16:20:45,931 DEBUG TestTest - Clean One
2011-07-29 16:20:45,931 DEBUG TestTest - Clean Two
2011-07-29 16:20:45,931 DEBUG TestTest - Clean Three
Tests run: 4, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 34.661 sec

A test failure in one data row means that all tests which depend on it be skipped for all subsequent data rows. This is not correct, because only the test method of the current data row should be skipped.

Also a weird behaviour is that first all test1() methods try to run, then test2(). (this can be seen if you comment out the Assert.fail() line.

We have test classes which represent complex scenarios, and the test methods are steps in these scenarios, thus dependencies between test methods are a usual thing.

[vadipp]

This is probably related to issue 71, but I decided to file a new issue.

[vadipp]

The "Comment and close" button was too close :)

[cbeust]

I'm aware of this limitation, this is how TestNG is implemented, right now: dependencies work on the class level, not on the instance level. When you depend on a method, you really depend on this method, not on "this method on instance xxx". TestNG currently doesn't make this distinction, hence the result you're seeing.

The only workaround right now is to call your methods together instead of using dependencies.

I'm considering ways to change this behavior but it's not trivial.

[cbeust]

This should be fixed in the beta, can you try it and report back? http://testng.org/beta

[vadipp]

I've just checked and it seems to work in the 6.1.2 beta. Thanks, Cedric!

I'm still concerned about the test method order, I'll report a separate bug.

This one is closed.