Introduce `@BeforeSuite` and `@AfterSuite` annotations

[ondrejlerch]

I suggest that @BeforeSuite and @AfterSuite annotations should be added to JUnit 5.

Obviously they should execute before and after the whole suite.

Or is there some other mechanism to accomplish that?

I suggest that there should not be static restriction, i.e. also non-static methods could be annotated with @BeforeSuite and @AfterSuite.

I personally prefer non-static as we use Spring in our tests and want to use injected beans in @BeforeSuite method.

[smoyer64]

Related to #163.

There's not always an easy way to control what's considered a "Suite". If you look at the Maven plugin, it controls the inclusion/exclusion of tests via XML. The JUnitPlatform test runner allows a suite to be defined declaratively but it's not honored by the Maven plugin.

If there was always a declarative way to run a suite, I'd say that @BeforeAll/@AfterAll could just be reused on the suite declaration. As things stand and in the name of unambiguous behavior, I prefer your annotations.

[ttddyy]

Related to #19 (e.g.: @BeforeAllTests/@AfterAllTests)

[ondrejlerch]

In my consideration "Suite" can be defined via:

• Maven/Gradle plugin inclusion/exclusion
• JUnitPlatform declarative declaration
• IDE when I click Run on the whole package of tests

In all those cases I would expect that @BeforeSuite/@AfterSuite would be executed just once before/after the whole bunch of tests, for example to perform time consuming operation such as database cleanup.

[sebersole]

I agree with everything @ondrejlerch brings up

[marcphilipp]

@ondrejlerch How would JUnit find @BeforeSuite/@AfterSuite annotated methods? For example, when you run a test class from an IDE would it only look at the test class or someplace else?

The Store abstraction along with the root ExtensionContext provides a means to prepare something once and share it between multiple test classes. However, there's no way to clean it up at the moment (that depends on #742).

[sebersole]

@marcphilipp you could consider a solution such as Jandex, which provides you an index over all annotations

[sormuras]

https://github.com/wildfly/jandex ?

[marcphilipp]

Well, in any case, it has to be obvious for test authors where JUnit will actually look for these annotations, i.e. if they will be applied when they execute a test class.

[sormuras]

environment (shell/build tool/IDE/...)  (1)
  platform/launcher (DefaultLauncher)   (2)
    engine a (Jupiter)                 (3a)
    ...
    engine x                           (3x)
    engine z                           (3z)

I see an easy way to implement an extension point for 3a -- which frames the all tests executed by the Jupiter engine with a "before the first" and "after the last" callback.

Pseudo-code for JupiterTestEngine:

@Override
public final void execute(ExecutionRequest request) {
    // TODO context = createExecutionContext(request);
    // try {
    // TODO callBeforeExecutionCallbacks(context);
    new HierarchicalTestExecutor<>(request, context).execute();
    // } finally {
    // TODO callAfterExecutionCallbacks(context);
    // }
}

[sebersole]

For the record, I actually would not use the annotation specifically. I am more interested in the corresponding Extension (especially AfterSuiteCallback).

I was just answering the general question of "finding annotations". That is what Jandex excels at, whether you want to use it or not...

[marcphilipp]

I’m fine with adding an extension point that uses the existing registration mechanisms.

That should work for everything except Maven Surefire... see #1113.

[sebersole]

As I would never use Maven ever again, I can live with that ;)

But yes, that is exactly what would work best for us:

@ExtendWith( MyAfterSuiteCallback.class )
...

[sormuras]

Started working on a PR that introduces BeforeExecutionCallback and AfterExecutionCallback...

[ondrejlerch]

@marcphilipp To your question: How would JUnit find @BeforeSuite/@AfterSuite annotated methods? For example, when you run a test class from an IDE would it only look at the test class or someplace else? I suggest answer: JUnit should search among all executed test classes, not someplace else. So if you execute just one test class (via IDE or Maven or Gradle) it should look only there. However, if you execute multiple test classes it should look at all of them and and run @BeforeSuite first.

If we use JUnit 5 with Spring then Spring Context should be initialized first, then @BeforeSuite should be executed, so that we can use Spring in @BeforeSuite. Obviously @AfterSuite should be executed before Spring Context is closed for the same reason.

BeforeExecutionCallback and AfterExecutionCallback sound also good.

Solution should work with IDE, Maven and Gradle as those are most typical tools for test execution. I agree with @sebersole that Maven is not cutting edge, however, if Maven Surefire would not work then please suggest what would work in Maven as it is widely used.

Thanks!

[sebersole]

@sormuras Thanks so much for starting on this.

[sbrannen]

I'm also fine with the idea of introducing new BeforeExecutionCallback and AfterExecutionCallback extension APIs.

But... we'll have to work on the names. 😉

That and the implementation and semantics of course.

[sbrannen]

Hi everybody,

I'd like to point out that we have to be very cautious here about the use of the term "suite" for a feature like this.

744 is the intended solution for suites on the Platform.

Whereas, this issue is about before and after callbacks that are specific to a given run of the Jupiter TestEngine.

So please keep this in mind so that we don't hinder the applicability and utility of whatever is eventually done for #744.

Thanks,

Sam

[sebersole]

Given that clarification, maybe BeforeTestEngineExecutionCallback and AfterTestEngineExecutionCallback?

[nipafx]

This StackOverflow question provides an interesting use case (although there may be better solutions than @BeforeSuite):

We are using Maven to run JUnit5 functional tests. Most of the tests require data to go through our entire system (a Storm topology) which could take around 1 minute. We need to send some data through and then wait 1 minute before we run the tests.

We need to stage the data and then communicate keys and such to the tests.

Is there a way to run a method/class to stage the data, then wait one minute, then run the rest of the tests?

[gortiz]

I have a case quite similar to the one named by @nicolaiparlog. I work very close to some SQL and NoSQL databases and testing these entities are usually quite hard. If you are using MySQL/Postgres/Mongo you have three options:

1. Running tests agains a mocked or embedded database. This can be acceptable, but you are not testing exactly what you want
2. All developers and your CI have access to required databases (maybe local installations or a shared remote one). In this case you must be very careful. Your tests must be prepared to be running concurrently (good look testing DDL sentences with MySQL) and they must leave, in any situation, the database on a correct state.
3. You start a new database instance each time you launch a test. You can do it with some libraries that automatically unzip and starts the database for you or you can start a Docker container with exactly what you need (including initial data, which is fantastic). You can start the database once before you run your suite or once per test method/class. The latter can be done using JUnit annotations (there are some alternatives, I have my own that starts a docker container using Junit extensions, but there are some others. The first can be done manually or with Gradle/Maven, but I would like to be able to create a JUnit extension that launches the container when the suite starts and removes it when it finishes.

[sormuras]

Running those use-cases in single class might be an option. Then you may make use of @BeforeAll and friends. If the single class grows too large, well, you may split it up and care about starting your "setup" only once.

[gortiz]

I'm running these cases on a single class right now, but once you want to split the class, before/after suite could really simplify the code (and I can distribute a JUnit plugin so my coworkers can easily reuse my code on their projects).

BTW: A modern machine can start an empty MySQL/Postgres/Mongo container in a couple of seconds, so it is not really bad to start one per class. But the faster it goes, the more frequently you can run our mini-integration tests.

[sbrannen]

Keep in mind that an extension that implements BeforeAllCallback and is registered globally via the ServiceLoader mechanism can make use of CloseableResource and the ExtentionContext.Store to achieve many suite-like behaviors.

See #1555 and the discussion on Gitter linked from there for further details.

[RomanIovlev]

Did you plan to introduce before/after test run annotation in JUni5?

[SergiiNik]

hi All!

Instead of @BeforeSute I used like this:

Add to your Base abstract class (I mean abstract class where you initialize your driver in setUpDriver() method) this part of code:

private static boolean started = false;
static{
    if (!started) {
        started = true;
        try {
            setUpDriver();  //method where you initialize your driver
        } catch (MalformedURLException e) {
        }
    }
}

And now, if your test classes will extends from Base abstract class -> setUpDriver() method will be executed before first @Test only ONE time per project run.

[jbduncan]

@SergiiNik Nice suggestion! However AFAICT, it only works when running tests one after another (which is the default behaviour) instead of in parallel.

By my understanding, when running in parallel, the code setUpDriver() may get called multiple times before started = true becomes visible to all threads running the tests, so the double-checked locking pattern with a Lock may be needed in this case. :)

[jbduncan]

...or even better, the Initialization-on-demand holder idiom.

[jbduncan]

...although, I think the holder idiom only works if setUpDriver() returns something... 🤔

[Legionivo]

@SergiiNik Nice suggestion! However AFAICT, it only works when running tests one after another (which is the default behaviour) instead of in parallel.

By my understanding, when running in parallel, the code setUpDriver() may get called multiple times before started = true becomes visible to all threads running the tests, so the double-checked locking pattern with a Lock may be needed in this case. :)

Hello @jbduncan , we have our own almost similar with @SergiiNik solution and it works perfect in parallel, we use the following Junit parameters: junit.jupiter.execution.parallel.enabled=true junit.jupiter.execution.parallel.mode.default=concurrent junit.jupiter.testinstance.lifecycle.default=per_class See my gist: https://gist.github.com/Legionivo/23313f5998284e25af9a4eb6570a6360

[IsmagilovQA]

Hi guys! Do we have a chance to get beforeSuite/afterSuite feature done in the near future? Or for some reason, it is postponed?

[theBull]

Hi all, I want to echo the curiosity around this potential feature as well. Any word on whether this is something on the roadmap?

[marcphilipp]

Not on the current roadmap. There's a workaround using ClosableResource, though (see https://github.com/junit-team/junit5/issues/456#issuecomment-416945159).

[wind57]

Keep in mind that an extension that implements BeforeAllCallback and is registered globally via the ServiceLoader mechanism can make use of CloseableResource and the ExtentionContext.Store to achieve many suite-like behaviors.

See #1555 and the discussion on Gitter linked from there for further details.

This is great, very much appreciate the hint. It has a few drawbacks though:

• every test goes via that extension. yes, we do filter them by some annotation obviously, but you are still paying for a conditional check in every other test.

  • as soon as you need more then 3 (imho) different Before/AfterSuite, that extension becomes cumbersome to maintain (this has hit us).

  • if tests are run in parallel and in BeforeAllCallback there has to be a single thing done (think an embedded DB started only once): there has to be a volatile check against the fact IFF the DB was already started. Add 3 different logical test suites needed and there are 3 volatile reads done.

These are the ones I found while implementing this pattern.

[marcphilipp]

If you use Store.getOrComputeIfAbsent it will do the check for you in a thread-safe manner.

[praveenreddynarala]

Hi Team, would it be fine to ask status on implementing Before/After Test Suite annotations in Junit5? I had followed all above instructions to do (workaround), however, nothing works in my case.

Problem Statement: We have a requirement to generate ExtentReport for our all tests in single report. Using above all suggestions, was able to generate class level report. Apart from, our tests (test classes) need some set of pre-test data to execute and flush out post run all test classes.

If someone has any working solution, kindly share.

Thank you.

[marcphilipp]

@praveenreddynarala Could you please share a few more details about what you tried? For creating a report, we generally recommend using a TestExecutionListener instead.

[praveenreddynarala]

@marcphilipp please find the way I have created Extent Report in our project..

Step 1: We have create one TestCallback class which implements BeforeAllCallback, AfterAllCallback, Before and AfterTestExecutionCallback Step 2: In beforeAll Callback method, I am creating ExtentReport instance Step 3: In beforeTestExecution Callback method, I am actually creating a test ExtentReport.createTest(SampleTestMethod) Step 4: afterTestExecution Callback method, logging all test results Step 5: afterAll Callback method, flushing the Extent report

With above steps, it is always creating Extent Report per Test Class. I had looked at thread (https://github.com/junit-team/junit5/issues/1555) and created custom extension to create Extent report instance only once (before test classes) using ExtensionContext.Store.CloseableResource. But it does not work when I run tests in parallel using Maven plugin.

Would it be good to know, is there a way I can initialise Extent Report before all Tests run and flush the report after all tests run.

Not sure/aware of using TestExecutionListener and what it does. How would I integrate Extent Report using this.

Please let me know if you would like to know any more information in regards.

[marcphilipp]

But it does not work when I run tests in parallel using Maven plugin.

There's nothing we can do here to solve that since Surefire spins up multiple VMs for that. You'd have to write a report per VM (maybe using ${surefire.forkNumber} as part of the file name) and create a Maven goal or some other post-processing step to merge them if you need a single report.

[vlsi]

marcphilipp : How would JUnit find @BeforeSuite/@AfterSuite annotated methods? For example, when you run a test class from an IDE would it only look at the test class or someplace else?

@marcphilipp , @sormuras , what do you think of discovering the suite as a transitive closure of the extension links?

For instance:

@ParticipatesInSuite("com.example.virtualsuite.name")
@ParticipatesInSuite(SuiteClass.class)
class VerifyE2E_TestCase1_... {
// test methods go here
}

@ParticipatesInSuite("com.example.virtualsuite.name") // e.g. just a symbolic name
@ParticipatesInSuite(E2ESuiteClass.class) // so the suite class could be autocompleted, and annotations could be placed there
class VerifyE2E_TestCase2_... {
// test methods go here
}

Then:

1) When the user launches ./gradlew test, Gradle would pass all test classes to JUnit. JUnit would find @ParticipatesInSuite and it would group tests into suites 2) When the user launches a single test from IDE, IDE would pass test class/method name to JUnit, it would find @ParticipatesInSuite annotation, and create a suite with a single class only 3) At no point the user can forget to add a test to an XML suite or something like that. The problem with suites that "enumerate test classes" is the users can add a test and forget to list it in the suite. Then JUnit won't execute the test, and everybody would think that is fine.

I believe that would be a better superset of the current "static suites" feature, and it seems to be very well in line with the current use of @ExtendWith when every test class declares its dependencies and features rather than something external assigns features to test classes.

---

The key value-added here is that the users could batch data population to the system under test which is essential in E2E testing scenarios when there's "per upload" overhead, so it might be much faster to gather all the required data from all the classes and upload it once in "before suite" rather than perform small uploads for each test.

---

The suggestion does not cover the case when the same set of classes needs to be executed multiple times with different settings. For instance "install Windows VM and launch all tests against it", then "install Linux and launch all tests against it". It might need something like SuiteFactory or it might be covered by E2ESuiteClass returning multiple "suite objects"

[marcphilipp]

Team decision: Investigate the feasibility of adding @BeforeSuite and @AfterSuite support in @Suite classes.

[YomnaZidan]

Hi @marcphilipp ,

Please feedback what is the alternative of BeforeSuite and AfterSuite , in order to create 1 consolidated Extend Report across all Test Classes . Same Issue Described by @praveenreddynarala

Regards , Yomna Zidan

[kishkk84]

@YomnaZidan @praveenreddynarala I am facing same issue as you both with extent report generation. Did you guys find any alternatives, if so can you please share. That would be a great help. Thanks 🙏

[Furrial]

@kishkk84 @YomnaZidan you can take a look at my solution implemented based on workaround using ClosableResource (see 456 comment)

[difflaender]

We have the same requirement. In JUnit 4, we were using an ExternalResource in a class that was running with a custom Suite implementation in order to set up a database structure for all tests that were going to be executed in the suite and to clean up the database afterwards. Knowing which classes are going to be executed by the Suite when these BeforeSuite and AfterSuite hooks are called would be pretty important to us, since we ask each of the executed test classes about what database contents the test requires. This allows us to set up the database just once in the beginning, instead of individually for each test, which results in a lot of saved time if multiple test classes have the same database requirements.

[nagkumar]

Related to #163.

There's not always an easy way to control what's considered a "Suite". If you look at the Maven plugin, it controls the inclusion/exclusion of tests via XML. The JUnitPlatform test runner allows a suite to be defined declaratively but it's not honored by the Maven plugin.

If there was always a declarative way to run a suite, I'd say that @BeforeAll/@AfterAll could just be reused on the suite declaration. As things stand and in the name of unambiguous behavior, I prefer your annotations.

Can we not follow what the definition of suite is as per TestNG.. https://testng.org/doc/documentation-main.html

@BeforeSuite @AfterSuite

[MaudeVanhengel]

@marcphilipp Is there any update to this? I'm using Junit 5 in combo with Cucumber and Spring Boot. My Suite class looks like this:

@Suite
@SelectPackages("features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "stepdefs")
public class FrontofficeSuiteRunner {

}

I've tried a solution of creating a custom extension class with BeforeAllCallback and then extending my suite with it, but no success. I'm finding no workaround that works for my use case.

The Suite runs all feature files in my features package and I need the BeforeAll method to do database cleanup before running the features..

[RainerGanss]

@MaudeVanhengel What we endet up doing is a

• spring bean
• with @PostConstruct and @PreDestroy methods for the BeforeSuite- and AfterSuite code
• has a test profile

We add the profile to the test execution and write our "BeforeAllCallback"-Code in the annoated methods. You could also use the InitializingBean or listen for spring application events (e.g. ContextStartedEvent) and execute code once the context is fully started.

The problem with the Callbacks is, that you don't have access to the spring application context to for example initialize the database using spring repositories.

WARNING: This solutions does not exactly the same as the here proposed BeforeSuite and AfterSuite annotations. The code in the bean is executed every time, the application context gets created/destroyed. So if for example a test method is annotated with springs @DirtiesContext annotation, the code will be executed again. In our case, that is actually exactly what we want. But it might not be the correct thing in your case. Also mocking the context with @MockBean will destroy/re-create the context. So use this with caution.