search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.21k stars 4.06k forks source link

JUnit 5 support #6681

Open BlackIsTheNewBlack opened 5 years ago

BlackIsTheNewBlack commented 5 years ago

ATTENTION! Please read and follow:

  • if this is a question about how to build / test / query / deploy using Bazel, ask it on StackOverflow instead: https://stackoverflow.com/questions/tagged/bazel
  • if this is a discussion starter, send it to bazel-discuss@googlegroups.com
  • if this is a bug or feature request, fill the form below as best as you can.

Description of the problem / feature request:

JUnit v5 (Jupiter) is a significant new version of JUnit, bringing many new features and changes.

Bazel should natively support JUnit v5 (Jupiter) tests. JUnit v5 annotations and assertions cannot be run in v4.

Feature requests: what underlying problem are you trying to solve with this feature?

"bazel test" should run for JUnit5 tests

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Replace this line with your answer.

What operating system are you running Bazel on?

Windows.

What's the output of bazel info release?

0.19.1

Have you found anything relevant by searching the web?

https://github.com/JeffreyFalgout/bazel-junit5.git It fails with all kind of errors.

Any other information, logs, or outputs that you want to share?

Will upload sample on request.

irengrig commented 5 years ago

P1 because I consider this important feature.

lberki commented 5 years ago

No can do, at least not with the urgency that P1 bugs need although I do agree that we should do this eventually.

jbduncan commented 5 years ago

Good news: The JUnit 5 team have a sample for using JUnit 5 in Bazel! :)

https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-bazel

jbduncan commented 5 years ago

Sure, it's not native, but it at least (should) fix the problem that the other existing example, https://github.com/JeffreyFalgout/bazel-junit5.git, doesn't work.

BlackIsTheNewBlack commented 5 years ago

Thank you @irengrig and @lberki Thanks @jbduncan for the tip

raz-canva commented 5 years ago

Any update on a native solution for this?

martindow commented 4 years ago

The JUnit team's Bazel starter project (https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-bazel) only appears to work with the Java 8 Bazel toolchain.

This works fine (as does not specifying java_toolchain): bazel test --java_toolchain=@bazel_tools//tools/jdk:toolchain_java8 //...

Specifying any other toolchain causes the ConsoleLauncher to run, but the test log shows that it doesn't find any tests (so it reports the tests as passing, even when there are failures): bazel test --java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 //... (It's the same problem with toolchain_java9 and toolchain_java10).

The log output shows that it ran, but found no tests:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test
-----------------------------------------------------------------------------
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
└─ JUnit Jupiter ✔
Test run finished after 27 ms
[         1 containers found      ]
[         0 containers skipped    ]
[         1 containers started    ]
[         0 containers aborted    ]
[         1 containers successful ]
[         0 containers failed     ]
[         0 tests found           ]
[         0 tests skipped         ]
[         0 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         0 tests failed          ]

Any clues as to what could be causing this? Maybe org.junit.platform.console.ConsoleLauncher's classpath doesn't have the test classes on it (they're declared in deps)?

martindow commented 4 years ago

Update on my issue above: This appears to be a problem with the homebrew installation of bazel, version 0.29.0-homebrew. I've since used bazelisk to install version 0.29.0 directly and it's running the tests as expected. I have no idea what the difference is between the two...

martindow commented 4 years ago

Might be useful to clarify for any other Bazel newbies like myself what the java_junit5_test rule in junit5.bzl (https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-starter-bazel) is doing under the hood. It:

  1. Declares the required JUnit5 Maven artifacts in WORKSPACE and
  2. Creates a java_test rule with the main class set to org.junit.platform.console.ConsoleLauncher.

If you're using the new maven_install rule (https://github.com/bazelbuild/rules_jvm_external) and want to inline these steps into your project, then the following changes will cause the sample project to execute:

Add the maven_install rule to WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
RULES_JVM_EXTERNAL_TAG = "2.10"
RULES_JVM_EXTERNAL_SHA = "1bbf2e48d07686707dd85357e9a94da775e1dbd7c464272b3664283c9c716d26"
http_archive(
    name = "rules_jvm_external",
    strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
    sha256 = RULES_JVM_EXTERNAL_SHA,
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)
load("@rules_jvm_external//:defs.bzl", "maven_install")
maven_install(
    artifacts = [
        "org.junit.jupiter:junit-jupiter-api:5.5.0",
        "org.junit.jupiter:junit-jupiter-engine:5.5.0",
        "org.junit.jupiter:junit-jupiter-params:5.5.0",
        "org.apiguardian:apiguardian-api:1.0.0",
        "org.opentest4j:opentest4j:1.1.1",
        "org.junit.platform:junit-platform-commons:1.5.0",
        "org.junit.platform:junit-platform-console:1.5.0",
        "org.junit.platform:junit-platform-engine:1.5.0",
        "org.junit.platform:junit-platform-launcher:1.5.0",
        "org.junit.platform:junit-platform-suite-api:1.5.0",
    ],
    repositories = [
        "http://central.maven.org/maven2",
    ],
)

Then change the test BUILD file to directly declare the java_test rule like this:

package(
    default_visibility = ["//src/test:__subpackages__"],
)

java_test(
    name = "junit5-jupiter-starter-bazel-test",
    srcs = glob([
        "*.java",
    ]),
    use_testrunner = False,
    main_class = "org.junit.platform.console.ConsoleLauncher",
    args = ["--select-package", "com.example.project"],
    deps = [
        "//src/main/java/com/example/project:junit5-jupiter-starter-bazel",
        "@maven//:org_junit_jupiter_junit_jupiter_api",
        "@maven//:org_junit_jupiter_junit_jupiter_engine",
        "@maven//:org_junit_jupiter_junit_jupiter_params",
        "@maven//:org_junit_platform_junit_platform_suite_api",
        "@maven//:org_apiguardian_apiguardian_api",
        "@maven//:org_opentest4j_opentest4j",
    ],
    runtime_deps = [
        "@maven//:org_junit_platform_junit_platform_commons",
        "@maven//:org_junit_platform_junit_platform_console",
        "@maven//:org_junit_platform_junit_platform_engine",
        "@maven//:org_junit_platform_junit_platform_launcher",
        "@maven//:org_junit_platform_junit_platform_suite_api",
    ],
)

Might be useful to someone...

guw commented 4 years ago

@martindow Any luck experimenting with --reports-dir argument for getting a nice report of tests for Jenkins (and other CI systems)?

PeterIzso commented 4 years ago

@lberki @iirina Are there any news on the native support?

unoexperto commented 4 years ago

For those who curious following example worked for me in bazel 2.x https://github.com/bmuschko/bazel-examples/tree/master/java/junit5-test

guw commented 4 years ago

@unoexperto I also found a solution in case you want to expose/collect the JUnit reports for Jenkins.

https://github.com/salesforce/bazel-maven-proxy/tree/master/tools/junit5

razfriman commented 4 years ago

Bump - Would be great to have first class support for this

asinbow commented 4 years ago

We have built a workaround for it: https://github.com/junit-team/junit5-samples/pull/133 Please take a look. A blog post: https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068

smicha commented 3 years ago

I know it is not exactly a bazel issue. Do we know if the junit folks are working on bringing junit5 support natively to bazel anytime soon .

MikhailTymchukFT commented 3 years ago

Any timeline for this feature? We know it's P3, but JUnit 4 becomes rather old.

fivetran-noopurbathija commented 3 years ago

What is the timeline for this issue? Its been open since 2018.

razfriman commented 3 years ago

Bump

chrismgrayftsinc commented 3 years ago

So I finally figured out why the behavior @martindow noticed above is happening (that is, no tests being run for toolchains other than java8). The reason is the version of java in the runfiles directory is lower than the version of java that compiled the classes. Running the test classes directly through ConsoleLauncher gives an error like Caused by: java.lang.UnsupportedClassVersionError: com/foo/bar/FooTest has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0.

This problem seems to exist in bazel versions < 4, but unfortunately the latest bazel docker image is 3.5.0 so it's still relevant to us, since we use the docker image in our CI.

The next question is whether it is possible to force the version of java in the runfiles directory to a higher version.

chrismgrayftsinc commented 3 years ago

For users of the 3.5.0 docker version, the following bazelrc works for me:

test --define=ABSOLUTE_JAVABASE=/usr/lib/jvm/11.29.3-ca-jdk11.0.2/reduced/
test --javabase=@bazel_tools//tools/jdk:absolute_javabase
test --host_javabase=@bazel_tools//tools/jdk:absolute_javabase
test --java_runtime_version=@bazel_tools//tools/jdk:absolute_javabase
test --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
test --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla

I put this in a file named .bazel-ci.rc and invoke the tests with bazel --bazelrc=.bazel-ci.rc test --test_output=errors //:foo-tests

shs96c commented 2 years ago

I have a PR open for rules_jvm_external that adds a junit5 runner and java_junit5_test: https://github.com/bazelbuild/rules_jvm_external/pull/629 This is based on what we've been using extensively at work, and has a number of advantages:

The major difference between this and the other approaches is that we don't depend on the ConsoleLauncher, so we have more control over how outputs are generated and placed, and how tests are run.

TRL-Official commented 2 years ago

I have a PR open for rules_jvm_external that adds a junit5 runner and java_junit5_test: bazelbuild/rules_jvm_external#629 This is based on what we've been using extensively at work, and has a number of advantages:

  • java_junit5_test is a drop-in replacement for java_test
  • Integrates with most of the features of the bazel test runner (eg. test filtering)
  • Works fine in the IJ plugin: the tests are just picked up as regular bazel tests, and stdout and stderr are captured properly
  • We handle System.exit being called and prevent that taking out the test runner

The major difference between this and the other approaches is that we don't depend on the ConsoleLauncher, so we have more control over how outputs are generated and placed, and how tests are run.

Hi @shs96c Thanks for this work. But I have one question. Your implementation of this JUnit5 Runner supports nested test classes? For example:

class SomeTest {
  @Nested
  @DisplayName("")
  class SomeFirstLevelNestedTest {
    @Nested
    @DisplayName("")
    class SomeSecondLevelNestedTest {
      @Nested
      @DisplayName("")
      class SomeThirdLevelNestedTest {
        // and so on
        // nesting of internal tests can be quite deep
        @Test
        void name() {
          Assertions.assertTrue(true);
        }
      }
    }
  }
}
TRL-Official commented 2 years ago

We have built a workaround for it: junit-team/junit5-samples#133 Please take a look. A blog post: https://flexport.engineering/connecting-bazel-and-junit5-by-transforming-arguments-46440c6ea068

Hi @asinbow
Thanks for this work. But I have one question. Your implementation of this JUnit5 Runner supports nested test classes? For example:

class SomeTest {
  @Nested
  @DisplayName("")
  class SomeFirstLevelNestedTest {
    @Nested
    @DisplayName("")
    class SomeSecondLevelNestedTest {
      @Nested
      @DisplayName("")
      class SomeThirdLevelNestedTest {
        // and so on
        // nesting of internal tests can be quite deep
        @Test
        void name() {
          Assertions.assertTrue(true);
        }
      }
    }
  }
}
amilamanoj commented 2 years ago

Hi @comius, Do you have any updates on this?

shs96c commented 2 years ago

There is a JUnit5 runner that ships as part of https://github.com/bazel-contrib/rules_jvm

github-actions[bot] commented 11 months ago

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

mkobit commented 11 months ago

Looking at some of the issues referencing this (like internal Google dependencies), it appears that this should eventually be added a native feature. Should this be unmarked as stale, then?

shs96c commented 11 months ago

I'd be more than happy if the junit5 runner from contrib_rules_jvm was moved into Bazel, but my feeling from the presentations that Google has been doing is that more and more things are being pushed out of Bazel itself and into rulesets. If that's the case, leaving this issue closed seems reasonable, since there's already an existing solution to the problem.

daniel-b2c2 commented 4 months ago

Can somebody clarify please:

shs96c commented 4 months ago

The recommended way is to use contrib_rules_jvm, which then provides two ways to run junit5 tests. The first is via a java_junit5_test, which is a direct replacement for java_test. The other way is to use java_test_suite with the runner attribute set to junit5. This macro takes a list of sources and generates a test target per test class.

In both cases, you need to make sure to add the required junit5 jars to the classpath using the deps attribute. You can use load("@contrib_rules_jvm//java:defs.bzl", "junit5_deps") to get a macro that can add them for you. eg:

java_test_suite(
    name = "small-tests",
    size = "small",
    srcs = glob(["*.java"]),
    deps = [
        ...
    ] + junit5_deps(),
)
marcphilipp commented 4 months ago

There's a sample project provided by the JUnit team that shows this in action: https://github.com/junit-team/junit5-samples/tree/main/junit5-jupiter-starter-bazel

daniel-b2c2 commented 4 months ago

Thanks for clarifying the current approach/state @shs96c & @marcphilipp.

As of 26th June 2024, from what I've seen:

On the developer usability side:

shs96c commented 4 months ago

@daniel-b2c2, if you have a reproducible test case that demonstrates the problem, filing an issue with contrib_rules_jvm for any problems would be very helpful.

The integration between IntelliJ and Bazel is via the JUnit XML file that's generated after the test run, and is handled by the IJ Bazel plugin, so there's a few moving pieces to deal with. Sadly, there are about as many slight variations on the format of this file as there are test runners. There's documentation about how we've designed and developed the format used by contrib_rules_jvm in the README.md here. Again, if there are improvements that we can and should make, please do file an issue.

daniel-b2c2 commented 6 days ago

if you have a reproducible test case that demonstrates the problem, filing an issue with contrib_rules_jvm for any problems would be very helpful.

Thanks, I will be posting a reproducible test case in the coming days.

Quick revisit of the recommendations above a few months later to check what the latest approach is.

So I've taken a look at the documentation recently vs the example in the repo, and they've diverged somewhat from the above. The docs here uses load("@contrib_rules_jvm//docs:stardoc-input.bzl", "java_test_suite") rather than load("@contrib_rules_jvm//java:defs.bzl", "JUNIT5_DEPS", "java_test_suite")

From looking at the source code, it looks like stardoc-input wraps the underlying rule, unclear why this extra indirection is required but it appears in the docs. Normally I would do this if I was preparing to make something private or internal....

I will need the junit5 deps, so my question is can I safely blend the two? In the end I've gone for the following:

load("@contrib_rules_jvm//java:defs.bzl", "JUNIT5_DEPS")
load("@contrib_rules_jvm//docs:stardoc-input.bzl", "java_test_suite")

this certainly compiles, but I'm unclear if it's best practice here to be mixing rules?

Also why does JUNIT5_DEPS not include the following:

artifact("org.junit.jupiter:junit-jupiter-api"),
        artifact("org.junit.jupiter:junit-jupiter-params"),

Finally JUNIT5_DEPS and junit5_deps() seem to be equivalent, is there one that is more appropriate to use than the other - is it a style issue?

The examples in the repo seems to append the above arteifacts manually, so I am having to do the same, though I'm unclear why they're not part of the JUNIT5_DEPS set.

shs96c commented 6 days ago

That's a snafu on our part, and comes down to how we generate the docs using stardoc. The load should still be from @contrib_rules_jvm//java:defs.bzl.

The artifacts included in the junit5_deps are the bare minimum required to successfully run a JUnit5 test. You're welcome to add more dependencies if they are needed.

Stylistically, I prefer junit5_deps() to JUNIT5_DEPS, but both are fine. We have both because we originally used the constant.

We should update the examples in the repo, and I'd happily review a PR that does so.