Coverage on macOS doesn't work with BAZEL_USE_CPP_ONLY_TOOLCHAIN

[phst]

Description of the problem / feature request:

When using the non-Xcode C++ toolchain (BAZEL_USE_CPP_ONLY_TOOLCHAIN=1), coverage generation doesn't seem to work at all.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

BUILD file:

cc_test(
    name = "test",
    srcs = ["test.cc"],
    deps = [":lib"],
)

cc_library(
    name = "lib",
    srcs = ["lib.cc"],
    hdrs = ["lib.h"],
)

lib.h:

extern void Function();

lib.cc:

#include "lib.h"
#include <iostream>
void Function() {
  std::cout << "hello world" << std::endl;
}

test.cc:

#include "lib.h"
int main() {
  Function();
}

Then, run

BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 bazel --nohome_rc --nosystem_rc --noworkspace_rc \
  --bazelrc=/dev/null coverage   --test_env=VERBOSE_COVERAGE=1 \
  --test_output=all --nocache_test_results //:test

No coverage instrumentation is generated, and the output contains an error message from gcov:

gcov: for the -o option: may not occur within a group! gcov: Unknown command line argument '-output'. Try: '/Library/Developer/CommandLineTools/usr/bin/gcov --help' gcov: Did you mean '-o'?

Apparently collect_cc_coverage.sh attempts to call gcov merge -output, which doesn't work - this should use llvm-cov instead or something else.

What operating system are you running Bazel on?

macOS 12.2.1

What's the output of bazel info release?

release 5.0.0-homebrew

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Nothing (not in a Git repository)

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/10457 is somewhat related (albeit a bit pessimistic; coverage on macOS does work in general), but doesn't seem to touch on this specific bug.

Any other information, logs, or outputs that you want to share?

test.log

[comius]

@c-mita Can you triage?

[uri-canva]

There's a BAZEL_USE_LLVM_NATIVE_COVERAGE enviroment variable you can try, not sure if it helps.

[phst]

There's a BAZEL_USE_LLVM_NATIVE_COVERAGE enviroment variable you can try, not sure if it helps.

That doesn't seem to have any effect in my case. It looks like this variable is only used to modify compilation flags (https://github.com/bazelbuild/bazel/blob/34ce6a23f5a2be58bb59661dd5fc9ab586ea1703/tools/cpp/unix_cc_configure.bzl#L265), but it should then also affect how the coverage merger is invoked.

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[phst]

I'm quite sure this is still happening. In fact, with Bazel 7 (after https://github.com/bazelbuild/apple_support/pull/113), the C++-only toolchain is enabled by default, so coverage support on macOS is now broken by default (when not using rules_apple).

[phst]

After experimenting a bit, I managed to at least create something that looks like a valid coverage.dat file on macOS with Bazel 7:

bazel coverage --test_env=COVERAGE_GCOV_PATH=/Library/Developer/CommandLineTools/usr/bin/llvm-profdata --test_env=BAZEL_USE_LLVM_NATIVE_COVERAGE=1 --test_env=LLVM_COV=/tmp/llvm-cov --experimental_generate_llvm_lcov -- //target

Here, /tmp/llvm-cov is a script like this:

#!/bin/bash
shopt -s -o pipefail
xcrun llvm-cov "$@" | sed -r 's#^SF:/.+/execroot/_main/#SF:#'

The system llvm-cov appears to always write absolute filenames, and the merger tool will filter them out because they are not listed literally in the instrumented files. Maybe the sed 's#/proc/self/cwd/## pipe in collect_cc_coverage.sh attempts to do something similar, but that doesn't work because the filename prefix isn't literally /proc/self/cwd/.

[keith]

Passing --cxxopt=-ffile-compilation-dir=. helps (as long as your compiler version supports it). At least 1 of the failure causes is the covered files don't exist with sandboxing and works in place of the /proc/self/cwd/ workaround

[keith]

BAZEL_USE_LLVM_NATIVE_COVERAGE shouldn't be necessary on macOS since the condition accounts for that:

https://github.com/bazelbuild/bazel/blob/d5577e63e5160d404cd667ed7886e6a07995cdfe/tools/cpp/unix_cc_configure.bzl#L254-L262

I don't see how bazel is supposed to be discovering llvm-cov and llvm-profdata, seems like there might need to be a fix there.

[phst]

BAZEL_USE_LLVM_NATIVE_COVERAGE shouldn't be necessary on macOS

ack, thanks

I don't see how bazel is supposed to be discovering llvm-cov and llvm-profdata, seems like there might need to be a fix there.

Maybe I'm missing something, but wouldn't that just be xcrun llvm-{cov,profdata}?

[keith]

Yea it could be, but right now the tools are wired through some environment variables

[phst]

Passing --cxxopt=-ffile-compilation-dir=. helps (as long as your compiler version supports it). At least 1 of the failure causes is the covered files don't exist with sandboxing and works in place of the /proc/self/cwd/ workaround

Yes, that works, thanks!

So to summarize, to make llvm-cov coverage work on macOS, the following .bazelrc lines serve as a workaround:

coverage --experimental_generate_llvm_lcov
coverage --test_env=COVERAGE_GCOV_PATH=/Library/Developer/CommandLineTools/usr/bin/llvm-profdata
coverage --test_env=LLVM_COV=/Library/Developer/CommandLineTools/usr/bin/llvm-cov
coverage --copt=-ffile-compilation-dir=.

[phst]

Looks like in the meantime more hacks have become necessary. Specifically:

coverage --repo_env=GCOV=/Library/Developer/CommandLineTools/usr/bin/llvm-profdata
coverage --repo_env=BAZEL_LLVM_COV=/Library/Developer/CommandLineTools/usr/bin/llvm-cov
coverage --repo_env=BAZEL_LLVM_PROFDATA=/Library/Developer/CommandLineTools/usr/bin/llvm-profdata
coverage --test_env=GENERATE_LLVM_LCOV=1

The --repo_env flags are necessary because otherwise the tool filenames from the local C++ toolchain will in some cases override the --test_env ones. GENERATE_LLVM_LCOV seems to be necessary for collect_cc_coverage.sh to actually generate LCOV instead of .profdata files.