Open jkschneider opened 7 years ago
I can imagine supporting something like
@ArgumentSource(...)
class MyTest {
@ParameterizedTest
void feature1() { ... }
@ParameterizedTest
void feature2() { ... }
}
Would that suit your needs?
I think that is a good solution.
Related issues: #871, #853
For real-life examples, see GaugeTest and most of the other tests in its package.
@jkschneider Do you have time to work on a PR?
If a parameter source annotation is placed on an individual method in a type that's been annotated like this would it make sense for the more specific annotation to take precedence?
Yes as a matter of determinism? I think in practice, this would not be the way I'd recommend folks structure their test.
Is this feature on the roadmap? I can't migrate to 5 without it.
...unless someone knows of a workaround.
@wrlyonsjr I flipped it around with Micrometer's TCK by defining an abstract class with my test methods, each of which takes an implementation of MeterRegistry
and for which there are many implementations. The RegistryResolver
extension injects the implementation into each method at any nesting level.
Then there is a corresponding test for each implementation of MeterRegistry
that extends this base class.
The approach has the disadvantage that you can't run the TCK abstract class from the IDE and execute the tests for all implementations, but this is the best I could do.
It seems like my problem could be solved with TestTemplate et al.
Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.
Let me share my experience with parameterized tests in the new JUnit. Hope that helps to clarify the uses cases and requirements for Parameterized classes.
I think that this feature will be universally useful, not for TCK only. I see its ultimate goal in bringing down the cost of defining multiple tests sharing the same parameters source. Currently, the following code is duplicated:
@MethodSource
)."[{index}] = {3}"
).Having to repeat that code discourages the users to write short, focused tests. On top of that, if developers do have a luxury of copy-pasting that code, reviewers and maintainers have to read all of it.
As an alternative, I've tried a dynamic test for a simple use case (little setup code, no teardown), but got both a personal impression and some reviews from colleagues that it's "overcomplicated".
A perfect use case for this feature, in my opinion, is the following:
@Parameter
(as in JUnit 4), or setup method (@BeforeEach
) arguments.@AfterEach
).I think supporting something like @Parameter
makes sense for fields and method parameters.
I'm out of ideas where you'd put formatting strings for parameters that are shared for all parameterized tests, though.
@marcphilipp , I can think of a class-level annotation with name
attribute, or, if users need more flexibility, let them provide an instance method returning a test description + a method-level annotation, or a reference to the method in the class-level annotation (e.g., @Parameterized(name="#testDescription")
).
If no one on the core team is planning to work on this issue soon, I may try to implement an MVP. I am new to the code base, and have a couple of questions about the requirements:
@ParameterizedTest
be supported in a class annotated with @ArgumentsSource
(@Test
, other @TestTemplate
s)? It might get tricky for one of the most compelling use cases for parameterized classes is extracting test template parameters to the fields, and if there is no extension to resolve the values of these fields, it won't work, will it?Shall the extension support a product of primary parameters (defined at the class level) and secondary parameters (defined at the method level, as in the current implementation), like this:
@CsvSource(/* primary parameters, injected into constructor/BeforeEach/fields */)
class FooTest {
/** Invoked for each parameter in the source specified at the class level */
@ParameterizedTest
void foo() { }
/**
* Invoked for the cartesian product of parameters from
* the source specified at the class level
* and parameters from the source at the method level.
*/
@ParameterizedTest
@ValueSource(/* secondary parameters for #bar only */)
void bar(int secondaryParameter) { }
}
?
I can think of a class-level annotation with
name
attribute, or, if users need more flexibility, let them provide an instance method returning a test description + a method-level annotation, or a reference to the method in the class-level annotation (e.g.,@Parameterized(name="#testDescription")
).
Maybe. Let's postpone that part until we've had some more time to ponder it.
Shall anything except
@ParameterizedTest
be supported in a class annotated with@ArgumentsSource
(@Test
, other@TestTemplate
s)? It might get tricky for one of the most compelling use cases for parameterized classes is extracting test template parameters to the fields, and if there is no extension to resolve the values of these fields, it won't work, will it?
How about we annotate fields/method parameters with @Parameter(index=0, optional=true)
(optional
should be false
by default) and inject null
for non-parameterized tests?
Shall the extension support a product of primary parameters (defined at the class level) and secondary parameters (defined at the method level, as in the current implementation)
I would expect that the arguments provided by the @ValueSource
would be appended to those provided by the @CsvSource
in your example, just like when both annotations would be declared on the method.
Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.
Actually, the requests I've been hearing are for "parameterized test class" support, not for sharing the same parameters across methods.
In other words, I've been hearing requests to be able to execute all tests within a given test class multiple times with different sets of parameters. When compared to the existing "test template" abstraction, this new feature would rely on a new "test class template" abstraction.
Do people think we can address the issues discussed thus far in this issue with such a "test class template" abstraction?
Or are these orthogonal concerns?
I'm also wondering whether adding something like container templates (cf. #871) would be clearer. If we had those, we could introduce s.th. like @ParameterizedContainer
:
@CsvSource(...)
@ParameterizedContainer(name = ...)
class SomeTestClass {
@Parameter(0)
private String foo;
@Test
void test() {
...
}
}
Ahhh.... yes... #871 is what I was looking for!
I've added comments to #871 accordingly.
How about we annotate fields/method parameters with
@Parameter(index=0, optional=true)
(optional should be false by default) and inject null for non-parameterized tests?
Won't that be hard to explain to users and force them to write complicated setup/teardown code, if they have any?
I would expect that the arguments provided by the
@ValueSource
would be appended to those provided by the@CsvSource
in your example, just like when both annotations would be declared on the method.
I see, perhaps, my example failed to convey that I expected different sets of parameters, e.g., @CsvSource({"John, Doe", "Mark, Twain")
& @ValueSource(ints = 18, 45)
.
I think a "test class template" will certainly address the simple use case above, and, likely, clearer, because:
@Test
s that will share both the class-level parameters and the setup/teardown code.@TestTemplate
s, to get a cartesian product of the sets of class parameters, and the parameters of the @TestTemplate
, as in example above.With that much flexibility, "test class templates" are likely to satisfy both simple and advanced or niche uses cases.
I think this might work for my use case #1456, but I'm not certain what the exact syntax would look like reading this issue.
but I'm not certain what the exact syntax would look like reading this issue.
Fret not: we also don't know what the syntax will look like. It's up for debate. 😉
Would it help to have a concrete example to start from?
I'm using JUnit 4 for Selenium tests. I want to run them in multiple browsers. I'm using test class parameterization to do it.
As I have it set up now, a test looks like this:
public class TeamTest extends TestBase {
public TeamTest(String browser) {
super(browser);
}
@Test
public void canCreateTeam() {
// ...
TestBase
looks like this:
@RunWith(Parameterized.class)
public abstract class TestBase {
protected RemoteWebDriver driver;
protected TestBase(String browser) {
driver = new DriverCreator().getDriver(browser);
}
@Parameters(name = "{0}")
public static Object[] data() {
var browserString = Common.getString(Prop.browsers);
return browserString.split(" *, *");
}
// ...
In TestBase
, I parse the browser string (which ultimately comes from a configuration file) and then pass that to the test class constructor, which creates browser instances from it.
In each test class, all I have to do is inherit from TestBase
and define the constructor. I don't have to decorate individual test methods at all (other than having @Test
on them, of course). The test names come out as TeamTest.canCreateTeam[chrome]
, TeamTest.canCreateTeam[firefox]
, etc.
So, what might the syntax for this look like if it were added to JUnit 5?
So, what might the syntax for this look like if it were added to JUnit 5?
I think it could look very similar to what @marcphilipp proposed here: https://github.com/junit-team/junit5/issues/878#issuecomment-354544841
So, perhaps something like the following:
@ParameterizedContainer(name = "{0}")
@MethodSource("data")
public abstract class TestBase {
@Parameter(0)
private String browser;
protected RemoteWebDriver driver;
@BeforeEach
void setUpDriver() {
this.driver = new DriverCreator().getDriver(browser);
}
static Object[] data() {
var browserString = Common.getString(Prop.browsers);
return browserString.split(" *, *");
}
// ...
}
My 2 cents: I like the flexibility that JUnit 5 provides in this context, but I think that the JUnit 4 parameterized tests paradigm still has a lot of value. The main use-case for it is this: have a test class that accepts the same parameter type/value for all tests. In other words, avoid copy/paste @ParameterizedTest
annotation on each method since it is implied from the class-level capability.
@Parameterized
@ValuesSource(...whatever...)
public class FooTest {
private final Foo testedValue;
public FooTest(Foo testedValue) {
this.testedValue = testedValue;
}
@Test
public void testX() { .... }
@Test
public void testY() { .... }
@Test
public void testZ() { .... }
...etc... - no need to repeat @ParameterizedTest on each method -
}
@lgoldstein, isn't your @Parameterized
proposal identical to the aforementioned @ParameterizedContainer
proposal?
Seems like it, a few more thing though
@Parameter
annotation should be allowed on constructor parameter(s) as well@Parameter
annotation present then behavior should be like JUnit 4 - i.e., vs. implicit positional parameters of the constructor @lgoldstein Thanks for your 2 cents! 😉
While I haven't made it explicit above, I agree that @ParameterizedContainer
should support constructor injection in the same manner we currently support method injection for @ParameterizedTest
.
@lgoldstein Thanks for your 2 cents! 😉
+1
While I haven't made it explicit above, I agree that
@ParameterizedContainer
should support constructor injection in the same manner we currently support method injection for@ParameterizedTest
.
+1
Yeah, my thought is that the Constructor should have an optional argument supplier. So that you could have multiple arguments to the Constructor and it runs for each set of those and then multiple even more for the methods so you might end up with two sets of arguments for the Constructor and two sets of arguments for a test method. Resulting in each test method being executed four times. And of course you can see how this could expand to many many times
On Tue, Jul 3, 2018, 9:31 AM Sam Brannen notifications@github.com wrote:
@lgoldstein https://github.com/lgoldstein Thanks for your 2 cents! 😉
+1
While I haven't made it explicit above https://github.com/junit-team/junit5/issues/878#issuecomment-354544841, I agree that @ParameterizedContainer should support constructor injection in the same manner we currently support method injection for @ParameterizedTest.
+1
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/junit-team/junit5/issues/878#issuecomment-402158198, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAVjaMUO6Ms26jlpR2J2gBPY0BKSE6Iks5uC3JOgaJpZM4N1t1a .
-- Caleb Cushing
I've had a thought on this, and possibly one I should encode in another ticket...
Parameterized tests are nicer for running tests in functions; syntactically, and log wise. However... one of the problems I've had with tests in the past is with long tests. Imagine I want to test security for 3 different Roles, that have overlapping permissions, and this is a selenium test, each "parameter" (the role) takes 5 minutes to run. As far as I know there is no way to say run the 2nd argument on the command line (the possible ticket to open) and even if there was... how would this then impact that. I really like the idea of being able to specify my argument on the command line, even if it has to be positional. If anyone thinks discussing this as a feature (and how it would impact this) further, feel free to let me know.
Has there been any solution provided for this yet? In experimental mode or whatever? I am migrating from junit4 to junit5 and we have loads of tests that use @RunWith(Parametrized) which are very hard to migrate at this moment without a lot of rework
Has there been any solution provided for this yet? In experimental mode or whatever?
The team is currently finishing up the 5.4 RC1 release with 5.4 GA to follow shortly thereafter.
This particular issue is scheduled for 5.5, likely to be implemented in conjunction with #871.
What is the current time plan for this feature (as it seems like it didn't make it into 5.5)
@tobiasdiez We don't have a roadmap with hard deadlines. We really want to support this but there's always something more important to do. Thus, I'm afraid we cannot commit to a specific date other than soon™️.
FWIW, here's an extremely hack-slash workaround that uses @TestFactory
as an intermediary:
@TestFactory
Stream<DynamicNode> testFooBar() throws Exception {
return TestingUtils.parameterizedClassTester("foo={0} bar={1}", FooBar.class,
Stream.of(Arguments.of(123, "abc"));
}
@lombok.RequiredArgsConstructor
static class FooBar {
private final int foo;
private final String bar;
@BeforeEach
void prepare() {
...
}
@Test
void test1() {
...
}
}
TestingUtils.parameterizedClassTester() implementation here.
@Test
, @BeforeEach
and @AfterEach
,@BeforeAll
or @AfterAll
. (Although @BeforeAll
could be replaced by the constructor.)static
. (Probably anything more robust is better suited with full-blown annotations)org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor
such as features surrounding @BeforeEach
and @AfterEach
and handling of exception during callbacksMessageFormat
to get the display name should be more like org.junit.jupiter.params.ParameterizedTestNameFormatter
's implementationhi! Just wonder, any news about this in 2020? Thanks.
hi! Just wonder, any news about this in 2020?
Please see https://github.com/junit-team/junit5/issues/878#issuecomment-546459081
At the moment I am (ab)using(?) @Parameterized.Parameters
to create Spring Boot Tests with all permutations of a set of profiles.
Each of the profiles exposes a certain set of configurations and beans and represents "a module".
This is very old code and I'm trying to tidy it up for Junit 5. Are there any prettier alternatives for this @sbrannen ? I looked at some Stackoverflow posts but none have a decent implementation (creating an Abstract class and putting each test seperately wasn't my preferred solution) for creating a sort of parametrized test for different spring boot profiles (maybe spring boot has some examples you know off?).
I am posting this here because it relied on constructor injection for the @Parameters
and I'm not sure how else I could set the profile
value to setup the TestContextManager
.
@WebAppConfiguration
@ContextConfiguration(
initializers = { SomeCustomMockServletContextInitializer.class}
)
@RunWith(Parameterized.class)
@ActiveProfiles(resolver = ITBootstrapWithAdditionalModules.CustomProfilesResolver.class)
public class ITBootstrapWithAdditionalModules
{
private final String profile;
private TestContextManager testContextManager;
public ITBootstrapWithAdditionalModules( String profile
) {
this.profile = profile;
}
@Parameterized.Parameters(name = "{index}: modules: {0}")
public static Collection primeNumbers() {
Set<Set<String>> powerset = Sets.powerSet( Sets.newHashSet( "with-ui", "with-data-store", "with-redis" ) );
Object[] parameters = new Object[powerset.size()];
final AtomicInteger i = new AtomicInteger();
powerset.stream().forEach( item -> {
parameters[i.get()] = new Object[] { StringUtils.join( item, "," ) };
i.incrementAndGet();
} );
return Arrays.asList( parameters );
}
@Before
public void setUpContext() throws Exception {
System.setProperty( AbstractEnvironment.ACTIVE_PROFILES_PROPERTY_NAME, profile );
this.testContextManager = new TestContextManager( getClass() );
this.testContextManager.prepareTestInstance( this );
}
@After
public void destroyContext() {
testContextManager.getTestContext().markApplicationContextDirty(
DirtiesContext.HierarchyMode.EXHAUSTIVE );
testContextManager.getTestContext().setAttribute(
DependencyInjectionTestExecutionListener.REINJECT_DEPENDENCIES_ATTRIBUTE, Boolean.TRUE );
System.clearProperty( AbstractEnvironment.ACTIVE_PROFILES_PROPERTY_NAME );
}
@Test
public void servicesShouldStartup() {
}
@Configuration
protected static class Config
{
@Profile("with-ui")
@Configuration
public static class UiModuleProfile
{
@Bean
public UiModule UiModule() {
return new UiModule();
}
@Bean
public BootstrapUiModule bootstrapUiModule() {
return new BootstrapUiModule();
}
}
@Profile("with-redis")
@Configuration
public static class RedisProfile
{
@Bean
public RedisModule redisModule() {
return new RedisModule();
}
}
@Profile("with-datastore")
@Configuration
public static class DataStoreProfile
{
@Bean
public DataStoreModule dataStoreModule() {
return new DataStoreModule();
}
}
}
static class CustomProfilesResolver implements ActiveProfilesResolver
{
public CustomProfilesResolver() {
}
@Override
public String[] resolve( Class<?> testClass ) {
return System.getProperty( AbstractEnvironment.ACTIVE_PROFILES_PROPERTY_NAME ).split( "," );
}
}
}
We have a very similar usecase, at the moment we are either forced to duplicate code (or use inheritance/...) or only test a single combination. Implementing this feature could greatly increase our coverage while keeping the tests readable and without duplicates.
My current project supplies a TCK that needs to run test classes a variable number of times: once for each feature module discovered on the API under test.
A few reflections based on what we've explored and landed on:
@ParameterizedTest
annotation on the class or constructor (in our case, with a MethodSource) would solve our needs well.
@ParameterizedTest
in an awkward way@ParameterizedTest
, especially in cases like ours where the parameters are just simple types like String. The framework knows which parameters it passed to the constructor when creating the test instance, so recreating the test instances should be straightforward.It's unbelievable. Remove the feature without a replacement and 4 years later reply "Sorry, we have more important stuff". 🤦
@maxim5 If it’s that important to you, perhaps you could contribute something or sponsor someone else to do so? The JUnit team don’t owe us anything. Unless we’re prepared to put our time or money where our mouth is, we should wait patiently and respect the tough prioritization calls that they have to make.
@maxim5 There is in fact a workable solution in the meantime. See Micrometer's TCK. It may not be perfectly ideal, but works pretty well. OpenRewrite's TCK also uses this technique.
Hi Junit5 Team,
Thank you for the hard work you all are doing! I need your help on one of the issues I am facing while upgrading from Junit3 to Junit5, if you already have solution for the problem I am facing then it would be great if you can direct me to the solution.
Here is what I am looking for ... I have a csv file with almost 30 columns and 100 rows(each row is a test case), first 20 columns are input data for the test cases which I will be using under @BeforeAll class and another 10 columns(expected results) I will be using under the test cases. So is it possible to read few columns under @BeforeAll and few under @ParameterizedTest by using csv file?
Thank you, Sadhu
Hi @sadhup! JUnit 5 user here.
I don't believe it's possible to split the reading of data across @BeforeAll
and @ParameterizedTest
sections like this. It sounds like you should do it all in one or more @ParameterizedTest
s.
The @CsvFileSource
section in the user guide might be what you're after.
If this doesn't shed any more light for you, try reading this baeldung.com article, raising a StackOverflow question, or raising a new issue. I believe this issue is about TCKs, rather than CSV-based data.
I hope this helps.
P.S. Do your ~20 inputs and ~10 outputs really belong with each other? I ask because that sounds like a lot! If they don't really depend on each other, then I'd gently encourage you to split the CSV file into a number of smaller ones first, so your tests are easier to read.
Hi @sadhup! JUnit 5 user here.
I don't believe it's possible to split the reading of data across
@BeforeAll
and@ParameterizedTest
sections like this. It sounds like you should do it all in one or more@ParameterizedTest
s.The
@CsvFileSource
section in the user guide might be what you're after.If this doesn't shed any more light for you, try reading this baeldung.com article, raising a StackOverflow question, or raising a new issue. I believe this issue is about TCKs, rather than CSV-based data.
I hope this helps.
P.S. Do your ~20 inputs and ~10 outputs really belong with each other? I ask because that sounds like a lot! If they don't really depend on each other, then I'd gently encourage you to split the CSV file into a number of smaller ones first, so your tests are easier to read.
Hi @jbduncan,
Thank you for your quick response! Yes, I am using @CsvFileSource like given below :
@ParameterizedTest @CsvFileSource(resources = "/SupportAPIs.csv", numLinesToSkip = 1) public void testSupportAPI(String col1, String col2, String col3 ... String col30){ }
Regarding the ~20 inputs and ~10 outputs : Yes, they are related to each other. Each line in a csv file is one test case, I am setting up data using those ~ 20 inputs for a test case, and other 10 values in the same row are expected under the API response. So that's why I want to read col1 ... col20 under @BeforeEach/@BeforeAll class to setup the data for testing. col21 ... col30 under @ParameterizedTest.
Actually I have many files like that, so if I split them then I have to maintain 2X files. But please let me know if I split my file then how can i use two files, one under @BeforeEach/@BeforeAll and one under @ParameterizedTest.
I really appreciate your response on this issue.
Thank you, sadhup
Sorry @sadhup, I'm not sure what to suggest now.
Are you trying to move the 30 parameters into a common place to reduce boilerplate?
If so, and if you're willing to migrate your CSV files to another data format, then junit-pioneer has @JsonFileSource
. This would allow you to turn your 30 parameters into two: an "input" object, and an "output" object.
StackOverflow is a better place for questions like this, anyway, so raise your question there if you're still stuck or unsatisfied.
Overview
Currently, the target of
@ParameterizedTest
is constrained to methods. When creating technology compatibility kits, it would be awesome to be able to apply this (or a similar annotation) to the test class so that all tests in that class are parameterized the same way.Proposal
Rather than:
Something like:
Related Issues
14
871
1141
3157