Request: support passing build options to the extractor

[matta]

On Linux, which typically defaults to the gcc toolchain using GNU libstdc++, the following suffices to convince bazel to use clang and LLVM libc++ instead:

CC=clang bazel build --copt=-stdlib=libc++ --linkopt=-stdlib=libc++ //...

That is a small hack that Bazel supports for convenience, but the "principled" way to use non-system toolchains is to set one up explicitly and select it with --config=<blah> something like this imagined configuration (notice the lack of CC=clang):

bazel build --config=llvm-libcxx //...

However, I'm having trouble getting @hedron_compile_commands//:refresh_all to use non-default build options.

A CC=clang bazel run @hedron_compile_commands//:refresh_all invocation does honor the CC=clang portion, but if I run CC=clang bazel run --copt=-stdlib=libc++ --linkopt=-stdlib=libc++ @hedron_compile_commands//:refresh_all it has no effect for the generated compile_commands.json.

Context: I'm experimenting with C++20 coroutines. For this to work with gcc, I must pass -fcoroutines. For this to work with clang, I must pass -stdlib=libc++. For a reasonable experience with clang tooling the compile_commands.json should be set up correctly.

Currently I've worked around this problem by hacking my workspace's .bazelrc temporarily when running @hedron_compile_commands//:refresh_all, with the script at the bottom.

It would be nice if the extractor supported passing build args to the extractor.

#!/usr/bin/bash
#
# See https://github.com/hedronvision/bazel-compile-commands-extractor
#
# Setting CC=clang will cause Bazel's toolchain configuration logic to use
# clang instead of gcc.  This is unecessary on systems where clang is the
# default compiler (e.g. MacOS and FreeBSD).
#
# Setting --copt=-stdlib=libc++ causes clang to use LLVM libc++ instead of
# GNU libstdc++.  This is necessary only on systems where libc++ is not
# already the default (e.g. MacOS and FreeBSD).  See
# https://libcxx.llvm.org/UsingLibcxx.html
# opts=(--copt=-stdlib=libc++ --linkopt=-stdlib=libc++)
#
# Note: instead of the .bazelrc hacks done hbelow, we'd like to simply run
# this:
#
# CC=clang bazel run \
#     --copt='-stdlib=libc++' \
#     --linkopt='-stdlib=libc++' \
#     @hedron_compile_commands//:refresh_all
#
# ...but that doesn't work.

if ! workspace=$(bazel info workspace); then
    echo 2>&1 "error: must be run in a Bazel workspace"
    exit 1
fi
cd $workspace

# Back up the user's original .bazelrc in prep for modification.
original_bazelrc=
if [[ -f .bazelrc ]]; then
    cp .bazelrc orig.bazelrc
    original_bazelrc=orig.bazelrc
fi

# Script cleanup: restore the original .bazelrc or delete the one we're
# about to create.
function cleanup {
    if [[ -n "$original_bazelrc" ]]; then
        mv "$original_bazelrc" .bazelrc
    else
        rm -f .bazelrc
    fi
}
trap cleanup EXIT

# The @hedron_compile_commands//:refresh_all rule seems to always use the
# default build configuration.  So, put -stdlib build options into the
# default config (temporarily).
echo "build --copt='-stdlib=libc++'" >> .bazelrc
echo "build --linkopt='-stdlib=libc++'" >> .bazelrc

# Run with CC=clang for reasons described above.
CC=clang bazel run "${opts[@]}" @hedron_compile_commands//:refresh_all

[cpsauer]

Hey, Matt! Thanks for sticking with the tool and for reaching out again. Super appreciate your contributions and help of others.

Passing custom build flags (even per target) is definitely something we try to support already and use every day, so there's a good chance that there's already an easy path for this to work for you!

Any chance the refresh_compile_commands rule would do the trick for you? It's carefully crafted to still work for target patterns, like //...

Lmk! Here to help if not. You might have to convert the environment variable into a --compiler flag. And if you're finding the docs deficient there, would love to make em better together.

Just to check in more broadly: Are things otherwise working okay for you on gcc/linux/emacs after your great PR?

Thanks! Chris

[cpsauer]

I'm going to close this (optimistically), but if that fix isn't working, lmk, and I'll reopen it and jump right back in with you.

[matta]

Thanks Chris, I had a chance to try this out today. I'm able to pass the --copt and --linkopt args, but not the CC environment variable change. My script is now a one-liner:

CC=clang bazel run @//:refresh_compile_commands

It would be nice to be able to pass arbitrary environment variables to the refresh_compile_commands() rule. The --compiler arg doesn't work. There is magic auto-configuration logic in Blaze for the CC environment variable -- it seems like that env var is essentially a configuration knob for Bazel's default "platform" compiler. When I attempt to set --compiler=clang or --compiler=clang++ I get this:

INFO: Build options --compiler, --copt, --host_compiler, and 1 more have changed, discarding analysis cache.
INFO: Analyzed target //:refresh_compile_commands (1 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:refresh_compile_commands up-to-date:
  bazel-bin/refresh_compile_commands.sh
INFO: Elapsed time: 0.052s, Critical Path: 0.00s
INFO: 2 processes: 2 internal.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
 >>> Analyzing commands used in @//...
ERROR: /home/matt/.cache/bazel/_bazel_matt/b06faae89ff42e5fae839cbe27f8d9e7/external/local_config_cc/BUILD:47:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'k8' and compiler 'clang'.
ERROR: /home/matt/.cache/bazel/_bazel_matt/b06faae89ff42e5fae839cbe27f8d9e7/external/local_config_cc/BUILD:47:19: Analysis of target '@local_config_cc//:toolchain' failed
ERROR: Analysis of target '//base:runexample' failed; build aborted:
 >>> Finished extracting commands for @//...

(this is a pretty low priority thing for me ... I'll be keeping my "refresh_compile_commands" script anyway, because I will never remember the blaze incantation, so I can set the env var there)

[cpsauer]

Wahoo! Delighted to hear it's working for you! "Incantation" is a good word for it.

If it's okay, I'm gonna punt on code changes around env vars for now. I need get a bunch of other stuff done for the startup I'm running. But also because this solution seems to work pretty well -- and because clangd doesn't support the next stage, which is passing them in compile commands. All that said, certainly not gonna stand in the way of a fix if it's worth it to you to write one. Will probably need to mesh with the windows changes coming down the pipe, though.

[cpsauer]

@matta, I added a new interface for passing flags from command line! I'd love to hear what you think] [Check the bazel run section of the README]

I'd also be curious if the environment variables part is still bugging you. (I might have thought they'd be inherited through.)