Preprocessor dependencies can come out wrong

[jpeach]

I'm using LLAMACC_LOCAL_CC=clang, LLAMACC_LOCAL_PREPROCESS=1 and LLAMACC_FULL_PREPROCESS=1.

CC command-line is:

2021-06-05 16:03:08.329101327+10:00 CLANG -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety \
-Wself-assign -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 \
-D_FORTIFY_SOURCE=1 -DNDEBUG -ffunction-sections -fdata-sections -std=c++0x \
-MD -MF bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/statusor.d \
-frandom-seed=bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/statusor.o \
-iquote external/com_google_protobuf -iquote bazel-out/host/bin/external/com_google_protobuf \
-isystem external/com_google_protobuf/src -isystem bazel-out/host/bin/external/com_google_protobuf/src \
-g0 -g0 -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function \
-Wno-write-strings -Wno-deprecated-declarations -no-canonical-prefixes -Wno-builtin-macro-redefined \
-D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" \
-c external/com_google_protobuf/src/google/protobuf/stubs/statusor.cc \
-o bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/statusor.o

Somehow the dependencies generated by llama get the leading /usr/ removed. For example, llama emits /bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/cstdio, but raw clang emits /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/cstdio.

Full dependencies are here https://gist.github.com/jpeach/1654065e0db6f99385e0bebe1159c86e

[jpeach]

I'll dig into this some more, see whether I can't come up with some more info.

[nelhage]

Huh. I'm not quite sure how this would explain it, but I do notice that on modern Ubuntu (at least?), /bin/ is a symlink to /usr/bin/, so those paths should actually refer to the same file. Is your system such a system? Does the result vary depending on whether you call clang from /usr/bin or /bin/?

[jpeach]

Ok, I think I understand this better now. There are some bazel caching effects going on, which made the results inconsistent in confusing ways.

When Bazel runs a sandboxed command, it resets the path, but it is not consistent about whether it passes down any pre-existing $PATH variable. For example:

PATH=/tmp/tmp.UZfJqPcrWw/bin:/home/jpeach/.cache/bazel/_bazel_jpeach/bca846880028b2eb76664b66797b90da/sandbox/linux-sandbox/4/execroot/build:/bin:/usr/bin:/usr/local/bin
...
PATH=/tmp/tmp.XvYPN5h5vi/bin:/home/jpeach/.cache/bazel/_bazel_jpeach/bca846880028b2eb76664b66797b90da/sandbox/linux-sandbox/1/execroot/build:/home/jpeach/.cache/bazelisk/downloads/bazelbuild/bazel-4.1.0-linux-x86_64/bin:/home/jpeach/bin:/home/jpeach/go/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/jpeach/bin:/home/jpeach/bin:/home/jpeach/go/bin:/home/jpeach/.cargo/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin

So depending on which $PATH we get, llamacc could invoke /bin/clang++ or /usr/bin/clang++.

Now, AFAICT Bazel ignores the CXX environment variable and just sends everything to CC. This also put me on the wrong track for a bit because I was expecting llama to always use LLAMACC_LOCAL_CC. It took me a little while to understand that llama will detect the language from the compilation flags and input file and use the appropriate compiler.

Now everything works \o/

[jpeach]

Closing, since this is not a llama problem.