Cannot use a @Factory(dataProvider) to generate tests in classes with @Test annotation

briangoetz commented 13 years ago

I'm trying to set up a test case which belongs to a group, but whose class body consists of a factory instead of test methods.

The "obvious' thing to do is:

@Test(groups={ "groupName' }) public class Foo { @Factory(dataProvider="d') public Object[] makeInstances(ArgType arg) { ... }

@DataProvider(name = 'd')
Object[][] fact() { ... }

}

But, TestNG complains in a confusing way:

test: [testng] [TestNG] [ERROR] [testng] Method makeInstances requires 1 parameters but 0 were supplied in the @Test annotation. [testng] The tests failed.

Here, it seems to be getting confused that makeInstances() should be a test method, even though it is marked as a factory. I can make the tests run by removing the @Test on the class, but then I can't put the test in a group.

I'm trying to do this all without using an XML file.

cbeust commented 13 years ago

Initial testing shows things working for me, can you email me a full example? I'm probably missing something you're doing.

By the way, note that you can use the shorter "factory + data provider + constructor" approach:

public class A { private Integer m_n;

@Factory(dataProvider = "d") public A(Integer n) { m_n = n; }

public A() {}

@DataProvider public Object[][] d() { return new Object[][] { { new Integer(42) }, { new Integer(43) } }; }

@Test public void a() { System.out.println("a() " + m_n); } }

briangoetz commented 13 years ago

By the way, note that you can use the shorter "factor + data provider + constructor" approach:

That occurred to me only later. Originally I thought I couldn't do it at all, because I wanted to create a variable multiple @Tests per row emitted by the data provider, but the shorter formulation is attractive enough that I can refactor into that.

Examples attached. package java.util;

import org.testng.annotations.Test;

import java.util.functions.Block;

/**

NullArgsTestHelpers */ public class NullArgsTestHelpers { public static Object[] makeNullArgTests(Fixture fix) { ArrayList
list = new ArrayList<>(); list.add(new NonNullArgsTest(fix)); for (int i=0; i<fix.args.length; i++) list.add(new NullArgsTest(fix, i)); return list.toArray(); }

public static class Fixture { public final Block<Object[]> block; public final Object[] args;
```
public Fixture(Block<Object[]> block, Object... args) {
  this.block = block;
  this.args = args;
}
```
}

public static class NonNullArgsTest { private final Fixture fix;
```
public NonNullArgsTest(Fixture fix) {
  this.fix = fix;
}

@Test
public void goodNonNull() {
  fix.block.apply(fix.args);
}
```
}

public static class NullArgsTest { private final Fixture fix; private final int whichArg;
```
public NullArgsTest(Fixture fix, int whichArg) {
  this.fix = fix;
  this.whichArg = whichArg;
}

@Test(expectedExceptions = NullPointerException.class)
public void throwWithNull() {
  Object[] args = Arrays.copyOf(fix.args, fix.args.length);
  args[whichArg] = null;
  fix.block.apply(args);
}
```
} }

package java.util;

import org.testng.annotations.DataProvider; import org.testng.annotations.Factory; import org.testng.annotations.Test;

import java.util.functions.Block; import java.util.functions.Mapper; import java.util.functions.Predicate; import java.util.functions.Reducer;

/**
- IteratorsNullTest -- tests of proper null handling in Iterators methods */ public class IteratorsNullTest extends LambdaTestCase {
  
  @Factory(dataProvider = "data") public Object[] makeIteratorsNullTests(NullArgsTestHelpers.Fixture fix) { return NullArgsTestHelpers.makeNullArgTests(fix); }
  
  @DataProvider(name = "data") public Object[][] makeData() { List list = upTo(10); return new Object[][] { { new NullArgsTestHelpers.Fixture(new Block<Object[]>() { public void apply(Object[] args) { Iterators.forEach((Iterator) args[0], (Block) args[1]); } }, list.iterator(), bEmpty) }, { new NullArgsTestHelpers.Fixture(new Block<Object[]>() { public void apply(Object[] args) { Iterators.filter((Iterator) args[0], (Predicate) args[1]); } }, list.iterator(), pTrue) }, { new NullArgsTestHelpers.Fixture(new Block<Object[]>() { public void apply(Object[] args) { Iterators.<Integer, Integer>map((Iterator) args[0], (Mapper<Integer, Integer>) args[1]); } }, list.iterator(), mDoubler) }, { new NullArgsTestHelpers.Fixture(new Block<Object[]>() { public void apply(Object[] args) { Iterators.<Integer, Integer>reduce((Iterator) args[0], 0, (Reducer<Integer,Integer>) args[1]); } }, list.iterator(), rPlus) }, { new NullArgsTestHelpers.Fixture(new Block<Object[]>() { public void apply(Object[] args) { Iterators.<Integer, Integer, Integer>mapReduce((Iterator) args[0], (Mapper<Integer, Integer>) args[1], 0, (Reducer<Integer,Integer>) args[2]); } }, list.iterator(), mDoubler, rPlus) } }; }; }
package java.util;

import java.util.concurrent.ThreadLocalRandom; import java.util.functions.*;

import static org.testng.Assert.assertEquals;

/**
- LambdaTestCase / public class LambdaTestCase { protected static Mapper<Integer, Integer> mDoubler = x -> x * 2; protected static IntMapper imDoubler = x -> x * 2; protected static LongMapper lmDoubler = x -> x * 2; protected static DoubleMapper dmDoubler = x -> x \ 2; protected static Predicate pFalse = x -> false; protected static Predicate pTrue = x -> true; protected static Predicate pEven = x -> 0 == x % 2; protected static Predicate pOdd = x -> 1 == x % 2; protected static Reducer<Integer, Integer> rPlus = (x, y) -> x+y; protected static Reducer<Integer, Integer> rMax = (x, y) -> Math.max(x,y); protected static Reducer<Integer, Integer> rMin = (x, y) -> Math.min(x,y); protected static IntIntReducer irPlus = (x, y) -> x+y; protected static LongLongReducer lrPlus = (x, y) -> x+y; protected static DoubleDoubleReducer drPlus = (x, y) -> x+y; protected static Block bEmpty = x -> { };
  
  protected ThreadLocalRandom random = ThreadLocalRandom.current();
  
  protected List upTo(int n) { ArrayList list = new ArrayList<>(); for (int i=1; i<=n; i++) { list.add(i); } return list; }
  
  protected List asDoubles(List integers) { ArrayList list = new ArrayList<>(); for (Integer i : integers) list.add((double) i); return list; }
  
  protected List asLongs(List integers) { ArrayList list = new ArrayList<>(); for (Integer i : integers) list.add((long) i); return list; }
  
  protected void assertCountSum(Iterable it, int count, int sum) { assertCountSum(it.iterator(), count, sum); }
  
  protected void assertCountSum(Iterator it, int count, int sum) { int c = 0; int s = 0; while (it.hasNext()) { int i = it.next(); c++; s += i; }
```
assertEquals(c, count);
assertEquals(s, sum);
```
  } }
cbeust commented 13 years ago

Good to see some closures, but I can't say that I've tested TestNG with Java 8 :-)

I'll try to bring your example down to something I can compile, see if you can do the same, maybe the issue is a Java 8 bug since I'm pretty confident in my data provider tests overall.
briangoetz commented 13 years ago
Its possible the problem could be java 8 instabilities. But, I found the solution (remove the @Test from the class) by reading this page:

http://groups.google.com/group/testng-users/browse_thread/thread/9339709067b5f24f/9845b39fa384c958?fwc=1

so I don't think it's that.

Here's Block:

public interface Block {

/** * Performs operations upon the provided object which may modify

that object * and/or external state. * @param t an input object / void apply(T t); }

The data provider returns a single Fixture argument per row; a Fixture has a Block<Object[]> and an Object[]. The helper code asserts that the block executes correctly when all the objects are substituted, and then changes each argument to null, one at a time, and asserts that an NPE results. You should be able to replace all the goop in the Fixture constructor with blocks that don't depend on lambdas, eliminate the "extends LambdaTestCase", and make it compile. Here's one that fails for me, without any lambdas:

@Test(groups = { "x" }) public class FooTest {

@Factory(dataProvider = "data") public Object[] makeIteratorsNullTests(NullArgsTestHelpers.Fixture

fix) { return NullArgsTestHelpers.makeNullArgTests(fix); }

@DataProvider(name = "data") public Object[][] makeData() { return new Object[][] { { new NullArgsTestHelpers.Fixture(new Block<Object[]>() { public void apply(Object[] args) { ((String) args[0]).length(); } }), "foo" } }; };

}

On 10/4/2011 3:26 AM, Cedric Beust wrote:

Good to see some closures, but I can't say that I've tested TestNG with Java 8 :-)

I'll try to bring your example down to something I can compile, see if you can do the same, maybe the issue is a Java 8 bug since I'm pretty confident in my data provider tests overall.
cbeust commented 13 years ago

I added a @Test at the class level but it's still passing for me so far, here is what I'm using:

@Test public class A { private Integer m_n;

@Factory(dataProvider = "d") public A(Integer n) { m_n = n; }

public A() {}

@DataProvider public Object[][] d() { return new Object[][] { { new Integer(42) }, { new Integer(43) } }; }

public void a() { System.out.println("a() " + m_n); } }

I'll try to expand this to look more like your example.

cbeust commented 13 years ago

Does my example above pass for you? If it does, can you expand it to make it break?
briangoetz commented 13 years ago
I got it down to a single independent file. If I take the @Test off the class, it works. The error I get is:

org.testng.TestNGException: Method ohNo requires 1 parameters but 0 were supplied in the @Test annotation. at org.testng.internal.Parameters.checkParameterTypes(Parameters.java:187) at org.testng.internal.Parameters.createParameters(Parameters.java:123) at org.testng.internal.Parameters.createParameters(Parameters.java:357) at org.testng.internal.Parameters.handleParameters(Parameters.java:434) at org.testng.internal.FactoryMethod.invoke(FactoryMethod.java:64) at org.testng.internal.TestNGClassFinder.(TestNGClassFinder.java:140) at org.testng.TestRunner.initMethods(TestRunner.java:406) at org.testng.TestRunner.init(TestRunner.java:232) at org.testng.TestRunner.init(TestRunner.java:202) at org.testng.TestRunner.(TestRunner.java:158) at org.testng.remote.RemoteTestNG$1.newTestRunner(RemoteTestNG.java:137) at org.testng.remote.RemoteTestNG$DelegatingTestRunnerFactory.newTestRunner(RemoteTestNG.java:266) at org.testng.SuiteRunner$ProxyTestRunnerFactory.newTestRunner(SuiteRunner.java:568) at org.testng.SuiteRunner.init(SuiteRunner.java:158) at org.testng.SuiteRunner.(SuiteRunner.java:112) at org.testng.TestNG.createSuiteRunner(TestNG.java:1246) at org.testng.TestNG.createSuiteRunners(TestNG.java:1233) at org.testng.TestNG.runSuitesLocally(TestNG.java:1085) at org.testng.TestNG.run(TestNG.java:1006) at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:107) at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:199) at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:170) at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Here's the test file:

import org.testng.annotations.DataProvider; import org.testng.annotations.Factory; import org.testng.annotations.Test;

import static org.testng.Assert.assertNotNull;

/**

FooTest */

@Test(groups = { "x" }) public class FooTest {

@Factory(dataProvider = "data") public Object[] ohNo(String s) { return makeNullArgTests(s); } public static class NullArgsTest { public final String s; public NullArgsTest(String s) { this.s = s; } @Test public void test() { assertNotNull(s); } } private Object[] makeNullArgTests(String s) { return new Object[0]; } @DataProvider(name = "data") public Object[][] makeData() { return new Object[][] { { "foo" } }; };

}

On 10/5/2011 2:49 AM, Cedric Beust wrote:

Does my example above pass for you? If it does, can you expand it to make it break?
cbeust commented 13 years ago

Thanks for the test case, I just fixed it. Interesting corner case you found :-)

Can you git pull and confirm the fix?

briangoetz commented 13 years ago

Building now. Might want to update the build instructions on http://testng.org/doc/download.html to include the bit about IVY libs. I did ant -lib . and it worked, but the less ANT-enabled might find themselves stuck (even though it is in README.build.)

Also the ANT file balks when building under a released version of Java 7, had to back off to Java 6.

All worked for me. (Though I had since decided to refactor to use the factory-on-constructor version, mostly because the pattern was more obvious for others to copy from.)

On 10/6/2011 1:35 AM, Cedric Beust wrote:

Thanks for the test case, I just fixed it. Interesting corner case you found :-)

Can you git pull and confirm the fix?
- © Githubissues.
- Githubissues is a development platform for aggregating issues.

testng-team / testng

Cannot use a @Factory(dataProvider) to generate tests in classes with @Test annotation #105