Test case grouping - Run subset of test cases depending on specific environment variables

[elshad-faire]

Hello,

We use JUnit 4 for our tests.

We have a use case that JUnit 4 does not support (at least we couldn't find it). We already built a custom JUnit test runner that achieves this but this requires us to annotate tests that we do not like. I was wondering if we could just contribute this to JUnit 4.

Here is the problem:

We have some test classes that take over 10 minutes to execute. These are neither rare nor easy to fix without involving the actual owners of the tests. So changing the test classes is not an option.

These test classes add to tail end latency a lot.

Our solution:

Run the test class multiple times where each time only a subset of test cases run. So altogether these runs still execute all the test cases but now we can explore parallel execution of this test class.

For example, assume in a gradle module :A, there is a test class ATest.kt which has test cases: test1, test2, test3, test4. Assume that breaking ATest.kt into two test case groups provides better results. Then the following 2 commands are executed in parallel:

TEST_CASE_GROUP_COUNT=2 TEST_CASE_GROUP_INDEX=0 ./gradlew :A:test –tests my.package.ATest TEST_CASE_GROUP_COUNT=2 TEST_CASE_GROUP_INDEX=1 ./gradlew :A:test –tests my.package.ATest

So all of ATest’s test cases are still executed. The difference now is more potential for parallelism and less tail end latency.

This solution worked really well for us. Would you be willing to accept this contribution? Let me know if you want to know more details.

Thanks.

[kcooney]

Couldn't you use Categories for this? If not, why not?

if not, you should be able to implement this as a Filter (or the Gradle JUnit plugin should be able to) without making changes to JUnit itself.

[elshad-faire]

Categories require tagging tests which means changing test files. I already have custom solution which requires changes to the test files. I would prefer not having to change the test files.

Do you have any example for how I can build such a Filter?

Btw, I already figured this out using JUnit 5. So I am going to just close this since I no longer need it.

[kcooney]

@elshad-faire rereading what the request was, this sounds like what Bazel calls test sharding. An implementation for JUnit4 is at https://github.com/bazelbuild/bazel/tree/master/src/java_tools/junitrunner/java/com/google/testing/junit/runner/sharding

No changes to the JUnit4 code were necessary to implement this.

The code leverages the JUnit Filter API. If any test classes use a runner that does not implement Filterable then you won't get fantastic parallelism, but most runners implement Filterable.

[elshad-faire]

Yes exactly this is bazel test sharding. The link you shared only contains the sharding part. It does not contain the test descriptor discovery part.

[kcooney]

The test list discovery part is just walking through the Description tree at the very beginning of the test run (to discover which tests are on the current "shard" so the other tests can be filtered out) and in the listener when the test run completes (to collect test results and determine if any were skipped due to the process receiving a SIGTERM if it ran too long).

Either way, no changes to JUnit we're needed to write that code. I know because I wrote the initial implementation. 😁

[elshad-faire]

This is similar to what I ended up doing using JUnit 5. I used custom PostDiscoveryFilter. In the first call PostDiscoveryFilter.apply, I discover the entire TestDescriptor tree. And then I decide which ones to filter out.

I was looking for a very simple way to do this. I did not want to deal with much customization :)

The code that bazel has, does it only work for junit 3?

[kcooney]

@elshad-faire it only supports JUnit 4. Bazel doesn't have built-in support for JUnit 5 (see https://github.com/bazelbuild/bazel/issues/6681)

[elshad-faire]

I see. Thanks for the info.