Disabling color with bazel test --nobuild causes an error

alexander-scott commented 2 years ago

Description of the problem / feature request:

When adding color=no to a bazel test --nobuild command, Bazel returns the following error message:

ERROR: Couldn't start the build. Unable to run tests

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

As far as I know this should be reproducible on any bazel project, e.g. the bazel examples.

First run without disabling color to see the expected result:

$ bazel test --nobuild //...
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 2 targets and 0 test targets...
INFO: Elapsed time: 0.154s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
INFO: Build completed successfully, 0 total actions

Then run with color disabled to see the bug:

$ bazel test --nobuild --color=no //...
Loading: 
Loading: 0 packages loaded
Analyzing: 2 targets (0 packages loaded, 0 targets configured)
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).

INFO: Found 2 targets and 0 test targets...

INFO: Elapsed time: 0.162s

INFO: 0 processes.

INFO: Build completed successfully, 0 total actions
ERROR: Couldn't start the build. Unable to run tests
INFO: Build completed successfully, 0 total actions

What operating system are you running Bazel on?

Ubuntu 20.04

What's the output of `bazel info release`?

release 4.1.0

philwo commented 2 years ago

That's very strange. I also observe that both cases produce an exit code of 1:

bazel test //examples/java-native/src/test/java/com/example/myproject:hello: No error message, exit code 0
bazel test --color=no //examples/java-native/src/test/java/com/example/myproject:hello: No error message, exit code 0
bazel test --nobuild //examples/java-native/src/test/java/com/example/myproject:hello: No error message, exit code 1
bazel test --nobuild --color=no //examples/java-native/src/test/java/com/example/myproject:hello: Error message, exit code 1

I always ran "bazel shutdown; bazel clean" between the attempts.

philwo commented 2 years ago

I'm not sure yet which one is the real bug:

That bazel test --nobuild does not show the error message. (In general the output seems to be pretty confusing even with --color=no).
Or that bazel test --nobuild fails, when it should work.

FaBrand commented 1 year ago

I encountered the same issue with the build --noshow_progress option.

With bazel test --nobuild the message ERROR: Couldn't start the build. Unable to run tests gets hidden.

However i can always get a very short glimpse of it when i run the command.

@philwo I would argue there are two: 1) The error message is probably being removed by some shell outputs. Bazel returns 1 but the error mesage is not printed in the console

2) bazel test --nobuild //some/test/targets/... fails when attempting to execute the test binaries that were not build beforehand.

However having this working could be quite useful.

Consider the use case to assert analysisphase for test targets only (with --build_tests_only) bazel test --nobuild //... could be used to prefetch some externals and seperate that in a retry mechanism to catch flaky infrastructure issues.

However bazel build --nobuild //... evaluates to a greater closure which can cause side-effects where select(...)'s are evaluated and can possibly fail.

It would also make the use case of failure testing with analysistests simpler where failing units-under-test would need to be tagged with '"manual"' or excluded otherwise for it to not fail with bazel test --build_tests_only //...

bazelbuild / bazel