Bad ordering of test methods when using a @Factory constructor with dataProvider

[vadipp]

I'm running this code on TestNG 6.1.2 beta:

import org.apache.log4j.Logger;
import org.testng.annotations.*;
import org.testng.annotations.Test;

public class TestTest
{
  private static Logger logger = Logger.getLogger( TestTest.class );
  private String param;

  @Factory( dataProvider = "prov" )
  public TestTest( String param )
  {
    this.param = param;
  }

  @DataProvider( name = "prov" )
  public static Object[][] dataProvider()
  {
    logger.debug( "Provide data" );
    return new Object[][] {
      { "One" },
      { "Two" },
      { "Three" },
    };
  }

  @BeforeClass
  public void prepare()
  {
    logger.debug( "Prepare " + param );
  }

  @Test
  public void test1()
  {
    logger.debug( "Test1 " + param );
  }

  @Test( dependsOnMethods = "test1" )
  public void test2()
  {
    logger.debug( "Test2 " + param );
  }

  @AfterClass
  public void clean()
  {
    logger.debug( "Clean " + param );
  }
}

I'm getting this result:

DEBUG TestTest - Prepare Two
DEBUG TestTest - Test1 Two
DEBUG TestTest - Prepare One
DEBUG TestTest - Test1 One
DEBUG TestTest - Prepare Three
DEBUG TestTest - Test1 Three
DEBUG TestTest - Test2 Three
DEBUG TestTest - Clean Three
DEBUG TestTest - Test2 One
DEBUG TestTest - Clean One
DEBUG TestTest - Test2 Two
DEBUG TestTest - Clean Two

I would better expect to get them in groups according to data rows I provided:

DEBUG TestTest - Prepare One
DEBUG TestTest - Test1 One
DEBUG TestTest - Test2 One
DEBUG TestTest - Clean One  
DEBUG TestTest - Prepare Two
DEBUG TestTest - Test1 Two
DEBUG TestTest - Test2 Two
DEBUG TestTest - Clean Two
DEBUG TestTest - Prepare Three
DEBUG TestTest - Test1 Three
DEBUG TestTest - Test2 Three
DEBUG TestTest - Clean Three

The order of groups (i.e. One, Two and Three) is not so significant, but the grouping itself would lead to better readability of results. Also, if the tests do some heavy-duty tasks in BeforeClass methods (e.g. create big objects), the ordering which I propose will guarantee that no more that one heavy-duty object persists at a given time (they will be deleted in corresponding AfterClass methods).

[cbeust]

I just fixed this in the beta, can you try and report back? http://testng.org/beta

Note: you have to use <suite group-by-instances="true"> for this to work (I'm considering deprecating this option and making it true all the time, but it's the case right now).

[vadipp]

Hello, Cedric!

Yes, it works! Now, finally, we are ready to start implementing nice data-driven testing in our project with TestNG :)

The only thing I have to ask is that you either add the new attribute to the DTD, or make the new behaviour the default. Please close this bug when you decide what to do (just not to forget).

Big thanks for your hard work!

[cbeust]

Great.

The published DTD already contains the new tag:

http://testng.org/testng-1.0.dtd.php