Introduce extension API for container templates

[nipafx]

Test templates are a great way to create extensions that generate test cases per method. If an extension wants to generate suites of tests per class, though, this is not easily possible. I propose to implement a container template extension point that functions similarly to test template:

• it defines the number of executions n - but in this case per container, which makes Jupiter work all its "execute tests in this class" magic not once but n times
• each invocation is called with a context, which most importantly contains the index of the current iteration
• additional extensions can be registered

It might make sense to consider whether other extensions should be allowed to easily access the container invocation context, maybe from the ExtensionContext they already receive. (Thinking about that, wouldn't it make sense to do the same for test templates, i.e. give ExtensionContext a method that returns the invocation context?)

Where parameterized test methods were the canonical example of how to use test templates, theories would be the canonical example for container templates. With this extension point the theories extension would use reflection to find out how many parameters were defined and then have the class be executed so many times, each time dropping a different set of parameters into all test methods:

• a = 0; b = 0
  • testAddition(a, b)
  • testSubtraction(a, b)
• a = 0; b = 1
  • testAddition(a, b)
  • testSubtraction(a, b)
• ...

Related Issues

• 878

• 1141

• 723

• 1239

• 2092

[marcphilipp]

Defining @ArgumentsSource on the test class (as proposed in #878) would be an equivalent solution, right?

[nipafx]

In parts. But it would still make the method the container for multiple tests instead of the class. And it would also require all methods to have the same parameters.

[sbrannen]

Spring users often ask for such capabilities -- for example, see SPR-16302.

[sbrannen]

Although SPR-16302 is a recent JIRA issue for spring, the topic has been raised multiple times over the years with regard to Spring's integration with JUnit 4. More recently, Spring users have begun to ask for such a feature in conjunction with JUnit Jupiter.

[sbrannen]

With regard to generic support for iteratively executing an entire container (test class), something like @RepeatedContainer would be very useful for frameworks such as Spring.

Thinking out loud...

Spring already has its own annotations for declaring things like active profiles, context configuration classes, etc. Thus, it would not make sense for a parameterized Spring test class to make use of annotations from Jupiter such as @ValueSource, etc. Rather, a framework like Spring would be more interested in knowing which iteration of the parameterized container is currently being executed so that Spring could then pick the correct configuration for that iteration.

Thoughts?

p.s. @RepeatedContainer could obviously be a specialization of @ContainerTemplate just like @RepeatedTest is a specialization of @TestTemplate.

[sbrannen]

Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.

[marcphilipp]

Another potential application: #723.

[sebersole]

Would @RepeatedContainer allow parameterization of the container itself? For us (Hibernate) this approach would be useful, but ideally we would be able to parameterize the container (class) itself, not each test.

[sormuras]

I don't think so. Look at how @RepeatedTest and @ParameterizedTest work. The former only counts an invocation counter up and executes the annotated method n times. The latter invocation count is predefined by the size of the Arguments passed to the annotated method.

Both do support default the ParameterResolver API. So, @RepeatedTest may be parameterized in a limited way.

[sbrannen]

Would @RepeatedContainer allow parameterization of the container itself? For us (Hibernate) this approach would be useful, but ideally we would be able to parameterize the container (class) itself, not each test.

Well, the main point of having container templates (where the container is a test class in this context) would in fact be to provide configuration that varies across invocations of the container. It's basically analogous to the existing support for test templates, just at a higher level.

So I don't see any reason why you couldn't "parameterize the container (class) itself".

Of course, if JUnit implements something like @RepeatedContainer out of the box, that support might not meet all of your customization needs, but built-in ParameterizedContainer support might suit your needs. And if not... it should be relatively easy to implement something like @RepeatedContainer on your own. Just take a look at the implementation for @RepeatedTest: it's very straightforward and only took me about half an hour to implement the core (though naturally longer to polish, test, and document).

[sebersole]

I just want to be able to run the same "set of tests" (test methods within a class) multiple times using slightly different configurations but ordered such that all tests within that class for a given configuration are run before the tests for that class for the next configuration.

JUnit has a number of mechanisms to help with those first parts already. However it has no support for controlling the order (unless we use dynamic tests that then have the drawbacks that we have been discussing in gitter).

So really I am open to any approach that ultimately works, but still not hearing that one "this is the way you should do that"...

This seemed like it would be a good approach. To me it makes sense that all the test-centric repetitions are ordered around the repeated test (at least as a default). And, to me anyway, by extension if I have a container-centric repetition it sure seems like (again, at least as the default) the executions should be ordered around the container.

But again, tell me how we should do this... The ParameterizedContainer you mention may oor may not work. It is hard to tell as there are literally 2 references to that on the world webs : this issue and another issue that links to this one :) So its really unclear what ParameterizedContainer will be to know if it would work :)

[sbrannen]

So its really unclear what ParameterizedContainer will be to know if it would work :)

True... it's usually difficult to predict the future.

But...

I just want to be able to run the same "set of tests" (test methods within a class) multiple times using slightly different configurations but ordered such that all tests within that class for a given configuration are run before the tests for that class for the next configuration.

AFAIK, that is exactly the plan for container templates and consequently for parameterized containers. At least, that's how I envision it working as well. Otherwise, IMHO, it would not be a very useful feature.

[sbrannen]

But again, tell me how we should do this...

Present: I think you're finding your own way at the moment (based on Gitter, etc.), using the tools currently available to you.

Future: I think container templates will make these types of use cases easier to implement and reason about.

[zhongr3n]

But again, tell me how we should do this...

Present: I think you're finding your own way at the moment (based on Gitter, etc.), using the tools currently available to you.

Could you share what tools currently available we can use to achieve class-level parameterization?

I implemented some solution with DynamicTest, but it quite verbose and doesn't support the lifecycle callbacks (pending https://github.com/junit-team/junit5/issues/378). Another alternative is defining the tests in an abstract class, and create subclass for each test parameter, but it's not very maintainable either.

[swaranga]

+1

I too have a use case to be able to repeat a parameterized test

[spganes]

+1. I also have test cases that need to repeat a parametrized test

[kasobol-msft]

+1 I also have a need to mix test templates. It would be great if JUnit has an option to generate cartesian product for arbitrary number of test templates. The idea (at least the outcome) in the https://github.com/junit-team/junit5/pull/2409 looks close to what I need - I'd like to develop a set of decoupled templates that I can mix in the tests. I understand that there are other means to achieve that like test factory, but they are quite verbose.

[nilic10]

+1

[Komdosh]

+1

[georgberky]

I would also love to have this feature because I need it whenever I am testing in branch by abstraction or decorator pattern scenarios where multiple implementations of the same interface need to have the same behavior - basically enforcing the Liskov Substitution Principle.

My current approach is to to make the test class abstract with a constructor for the interface and adding one subclass of the test class per interface implementation:

abstract class ThingTest {
    protected final Thing thing;

    protected ThingTest(Thing thing) {
        this.thing = thing;
    }

    class WithFirstImplementation extends ThingTest {
        public WithFirstImplementation() {
            super(new FirstImplementation());
        }
    }

    class WithSecondImplementation extends ThingTest {
        public WithSecondImplementation() {
            super(new SecondImplementation());
        }
    }

    //@Test follow here
}

It's a bit less boilerplate in Kotlin but noisy nevertheless. I've checked all current Resolvers and Extensions I could find, but there seems to be none that allows me to essentially return two instances of the test class but with fields replaced. Also MethodSource and other ParameterResolvers don't seem to work at the class level.

[ok2c]

@georgberky I ended up doing exactly the same, abstract test classes with nested concrete implementations, for Apache HttpClient. It gets the job done but is ugly as hell.

[whiskeysierra]

@georgberky That's exactly my use case as well - in fact I believe it's the textbook case how one would enforce LSP.

[nipafx]

@georgberky I think that approach is valid. I use it myself with some differences:

interface ThingTest {

    Thing getThingWithPropertyX();

    Thing getThingWithPropertyY();

    // @Test on default methods follow here

}

class SomeThingTest implements ThingTest {

    // implement factory methods

}

class AnotherThingTest implements ThingTest {

    // implement factory methods

}

Putting the tests on an interface like ThingTest opens up the possibility to more easily combine multiple interface tests in the same test class (e.g. SomeThingTest could implement another test interface) but if that doesn't work, the inner test class approach you describe is always an option. I find it's often necessary to test instances with different properties, which I express with different factory methods that may even have parameters.

Most importantly, though, I would not put the tests for the implementations on the interface test class/interface (ThingTest here) because I don't think that's where they belong. These tests apply to specific implementations (like SomeThing and AnotherThing) and should hence be attached to their test class(es). If you do it that way, the need to parameterize the test goes away.

[whiskeysierra]

Most importantly, though, I would not put the tests for the implementations on the interface test class/interface (ThingTest here) because I don't think that's where they belong.

Are you referring to implementation-specific tests? E.g. any test that is specific to SomeThing goes to SomeThingTest but any test that is general (=applies to all implementations of) Thing goes towards ThingTest?

As a separate, unrelated note: This pattern of separating tests from the concrete test classes has the additional benefit of properly separating test infrastructure code (bootstrapping a db, a file system, a memory data structure, any kind of dependency, or mock, etc.) from the actual tests.

[georgberky]

Hi @nipafx @whiskeysierra ,

thanks for posting. I like your approach. In effect it does the same thing as I'm doing right now. I have two concrete tests that run both the interface tests and the implementation-specific tests for the target implementation class.

The point of my comment here is less describing how I work around things but that I want to get rid of that workaround.

What I want to achieve using JUnit is that I can run the same set of tests through the interface but against multiple implementations of it. Think of it as

interface Thing {
    String operation();
}

class ThingImpl1 implements Thing {
    … 
}

class ThingImpl2 implements Thing{
    … 
}

class ThingTest {
    @ParameterizedTest
    @MethodSource("factoryMethod")
    private final Thing thing;

    @Test
    void someTest() {
         assertThat(thing.operation()).isEqualTo("expected");
    }

    public static List<Thing> factoryMethod() {
        return List.of(new ThingImpl1(), new ThingImpl2());
    }

I want someTest() to be run twice in this case, once with a test instance that has an instance of ThingImpl1 in its field and once with ThingImpl2 there.

I would also then move implementation-specific tests (e.g. Hibernate error handling) into their own test-classes. I want them clearly separated from the tests for the abstract contract offered by the interface. Right now that separation is sufficiently achieved through superclass/interface. It's not where I want to stay though.

[georgberky]

@nipafx I think I get what you mean now. I don't always put the subclasses into the abstract class. That depends on the amount of implementation-specific tests I have. In the example above I didn't have any, so it reads nicely like this. If I have plenty of implementation-specific tests, they go into their own class + file outside the superclass/interface.

[svschouw-bb]

We have a different use case for container based parameterization, maybe similar to https://github.com/junit-team/junit5/issues/871#issuecomment-396943044: we want to run a number of tests on multiple different environments, and setting up the environments takes a very long time (think minutes). So we don't want to set up the environment for every individual test.

Ideally this would be scoped even larger than test classes: running test suites per environment would be ideal, but just running all tests in a class per environment would be acceptable. Currently our tests are JUnit 4 tests with custom Runners.

[jbduncan]

@svschouw-bb You might want to consider the resources extension in JUnit Pioneer, which I authored recently. You'd need to tell it how to setup and tear down your environments, but in return it will call the setup/teardown code for you once per environment, and it will allow you to share environments between your tests.

But if you need to run a specific test method multiple times, once per environment, whilst still sharing environments across test classes, and your number of environments is fixed, then you could duplicate the test method one per environment and pass a different shared environment to each one...

[jbduncan]

Alternatively, a custom LauncherSessionListener could work.

[svschouw-bb]

@svschouw-bb You might want to consider the resources extension in JUnit Pioneer, which I authored recently. You'd need to tell it how to setup and tear down your environments, but in return it will call the setup/teardown code for you once per environment, and it will allow you to share environments between your tests.

But if you need to run a specific test method multiple times, once per environment, whilst still sharing environments across test classes, and your number of environments is fixed, then you could duplicate the test method one per environment and pass a different shared environment to each one...

Unfortunately the number of environments is dynamic and runs in the many dozens, of which about 8 can be active in parallel.

[sbrannen]

Update

The current goals for this feature are briefly outlined in https://github.com/junit-team/junit5/issues/878#issuecomment-1509873703.

Specifically, the current plan is to introduce at the least the following.

• @ContainerTemplate
• ContainerTemplateInvocationContextProvider extension API
• ContainerTemplateInvocationContext

[vlsi]

The lack of lifecycle methods like @BeforeEach in @ParameterizedTest looks like a showstopper for migrating to JUnit5. For instance, in pgjdbc we have quite a few tests where we initialize database connection in @Before method, and the connection properties are defined with test parameters (see https://github.com/search?q=repo%3Apgjdbc%2Fpgjdbc%20RunWith(Parameterized&type=code )

Unfortunately, JUnit5 does not provide access to the test parameters within before methods, so moving to JUnit5 is problematic.

Having both JUnit4 and JUnit5 classes on the classpath is problematic as it creates havoc of tests in different styles. At the same time, JUnit4 and JUnit5 have incompatible Assume exceptions, so test must select the right "assume" depending on JUnit version.

---

A year ago @sbrannen said the team decision was to add two classes and an annotation. Does anybody work on that? It the task up for grabs?

Of course, I understand there's a bit more work besides adding a couple of classes, however, the lack of lifecycle support for parameterized tests does impact users.

[sbrannen]

At the same time, JUnit4 and JUnit5 have incompatible Assume exceptions, so test must select the right "assume" depending on JUnit version.

It's true that JUnit 4 does not support JUnit Jupiter's exception for failed assumptions.

However, JUnit Jupiter actually supports JUnit 4's exception for failed assumptions.

Granted, that does not eliminate the issue, but it does help a little.

[sbrannen]

A year ago @sbrannen said the team decision was to add two classes and an annotation. Does anybody work on that? It the task up for grabs?

No, we do not put issues of this nature "up for grabs" due to the complexity and impact on the entire internals of JUnit Jupiter.

Of course, I understand there's a bit more work besides adding a couple of classes,

Yes, that's an understatement. 😇

The complexity of the implementation is actually what has kept us from tackling this one. The simple (unfortunate) fact is that no one in the core team has had the time to take on this issue.

In other words, this is likely something that will require weeks of work to implement, and that does not account for tests, documentation, interoperability with third-party frameworks, etc.

however, the lack of lifecycle support for parameterized tests does impact users.

We are aware of such issues and truly hope to make progress on this issue during the course of 2024.