unnecessary stubbings issues are swallowed when tests run with Mockito

[henrypoon]

Description of the bug:

The following two issues are found when I have unnecessary stubbing in tests with Mockito and JUnit.

1. unnecessary stubbings errors are swallowed and test passed when test_filter is being used, however, it failed as expected without test_filter
2. unnecessary stubbings errors are NOT reported in JUnit reports regardless of using test_filter or not. The JUnit report shows all tests have passed.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Add some unnecessary stubbings in a test
2. Run bazel run with the corresponding target and it would fail as expected, however, the JUnit report doesn't mention this at all

3. Run bazel run with the corresponding target as well as test_filter for a specific test case, it wouldn't fail at all and the JUnit report doesn't mention this error either.

   Please follow this repo to reproduce.

Which operating system are you running Bazel on?

MacOS Monterey

What is the output of bazel info release?

release 5.2.0-homebrew

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

Also reported to https://github.com/bazelbuild/rules_java/issues/71

[sgowroji]

Hello @henrypoon, Above repo still fails with this

bazel test //:test --test_filter=TestGreet

. Added error logs below

(base) sgowroji-macbookpro:bazel-test-filter-issues sgowroji$ bazel test //:test --test_filter=TestGreet
INFO: Analyzed target //:test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //:test (see /private/var/tmp/_bazel_sgowroji/3d50e8ad2cc48afbd50aafcb247a4ca4/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/test.log)
Target //:test up-to-date:
  bazel-bin/test.jar
  bazel-bin/test
INFO: Elapsed time: 0.562s, Critical Path: 0.39s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//:test                                                                  FAILED in 0.2s
  /private/var/tmp/_bazel_sgowroji/3d50e8ad2cc48afbd50aafcb247a4ca4/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

[henrypoon]

Hello @henrypoon, Above repo still fails with this

bazel test //:test --test_filter=TestGreet

. Added error logs below

(base) sgowroji-macbookpro:bazel-test-filter-issues sgowroji$ bazel test //:test --test_filter=TestGreet
INFO: Analyzed target //:test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //:test (see /private/var/tmp/_bazel_sgowroji/3d50e8ad2cc48afbd50aafcb247a4ca4/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/test.log)
Target //:test up-to-date:
  bazel-bin/test.jar
  bazel-bin/test
INFO: Elapsed time: 0.562s, Critical Path: 0.39s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//:test                                                                  FAILED in 0.2s
  /private/var/tmp/_bazel_sgowroji/3d50e8ad2cc48afbd50aafcb247a4ca4/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

May I ask what errors are shown from your local? As in from the test.log

[sgowroji]

Sorry for the confusion. It was security.manager exceptions resolved by test arguments. Shared your request to the right team who can assist you on the above. Thanks!

[henrypoon]

UPDATE Explicitly setting the strict rule worked as expected and can solve the issue temporarily.

@Rule
public MockitoRule mockito = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

However, I think it works differently as it's like validating the JUnit rule on a class/test level. Mockito runner should already enable strict mode by default according to their API https://www.javadoc.io/doc/org.mockito/mockito-core/latest/org/mockito/junit/MockitoJUnitRunner.html.

[gmishkin]

Please see https://github.com/bazelbuild/bazel/pull/15023

For the --test_filter issue, that's a Mockito thing, it's because of this but I haven't figured a good way to avoid it.

[jmmarsss]

Context: I work with @henrypoon We did later find out that MockitoJunitRunner would not conduct a stub validation if the tests are filtered, in other words, because of how bazel chains all the tests together, hence running tests with --test_filter would not trigger that..

You can refer this to force test failing when running with --test-filter, if that is what you want...

@Rule
public MockitoRule mockito = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

Hopefully your pr can get merged soon... @gmishkin

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[ckilian867]

Can confirm that we are still running into this issue in 7.0.2. Would like for this not to be closed as stale

[timothyg-stripe]

I think this is a duplicate of https://github.com/bazelbuild/bazel/issues/19949 (at least in the case where --test_filter is not specified)