Closed oakad closed 1 year ago
A full dump of the failure:
SUBCOMMAND: # @boringssl//:crypto [action 'Compiling src/crypto/cipher_extra/e_rc2.c', configuration: e530bf6bc4206cb6b15d1b7417fef783495fdc598b0acff74769e88b96f50e3a, execution platform: @local_config_platform//:host]
(cd /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/execroot/myproj-client && \
exec env - \
PATH=/opt/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/home/oakad/.dotnet/tools:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
PWD=/proc/self/cwd \
external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -gcc-toolchain external/androidndk/ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -target x86_64-none-linux-android -fPIC -isystemexternal/androidndk/ndk/sysroot/usr/include/x86_64-linux-android '-D__ANDROID_API__=29' -no-canonical-prefixes -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -funwind-tables -fstack-protector-strong -fno-addrsig '-Werror=return-type' '-Werror=int-to-pointer-cast' '-Werror=pointer-to-int-cast' '-Werror=implicit-function-declaration' -O0 -g -MD -MF bazel-out/android-x86_64-dbg/bin/external/boringssl/_objs/crypto/e_rc2.pic.d '-frandom-seed=bazel-out/android-x86_64-dbg/bin/external/boringssl/_objs/crypto/e_rc2.pic.o' -fPIC -iquote external/boringssl -iquote bazel-out/android-x86_64-dbg/bin/external/boringssl -isystem external/boringssl/src/include -isystem bazel-out/android-x86_64-dbg/bin/external/boringssl/src/include -DOPENSSL_NO_ASM '--sysroot=external/androidndk/ndk/platforms/android-29/arch-x86_64' -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem external/androidndk/ndk/sources/android/support/include -isystemexternal/androidndk/ndk/sysroot/usr/include -c external/boringssl/src/crypto/cipher_extra/e_rc2.c -o bazel-out/android-x86_64-dbg/bin/external/boringssl/_objs/crypto/e_rc2.pic.o)
# Configuration: e530bf6bc4206cb6b15d1b7417fef783495fdc598b0acff74769e88b96f50e3a
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # @com_github_cares_cares//:ares [action 'Compiling bitncmp.c', configuration: e530bf6bc4206cb6b15d1b7417fef783495fdc598b0acff74769e88b96f50e3a, execution platform: @local_config_platform//:host]
(cd /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/execroot/myproj-client && \
exec env - \
PATH=/opt/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/home/oakad/.dotnet/tools:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
PWD=/proc/self/cwd \
external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -gcc-toolchain external/androidndk/ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -target x86_64-none-linux-android -fPIC -isystemexternal/androidndk/ndk/sysroot/usr/include/x86_64-linux-android '-D__ANDROID_API__=29' -no-canonical-prefixes -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -funwind-tables -fstack-protector-strong -fno-addrsig '-Werror=return-type' '-Werror=int-to-pointer-cast' '-Werror=pointer-to-int-cast' '-Werror=implicit-function-declaration' -O0 -g -MD -MF bazel-out/android-x86_64-dbg/bin/external/com_github_cares_cares/_objs/ares/bitncmp.pic.d '-frandom-seed=bazel-out/android-x86_64-dbg/bin/external/com_github_cares_cares/_objs/ares/bitncmp.pic.o' -fPIC -DCARES_STATICLIB -iquote external/com_github_cares_cares -iquote bazel-out/android-x86_64-dbg/bin/external/com_github_cares_cares -isystem external/com_github_cares_cares -isystem bazel-out/android-x86_64-dbg/bin/external/com_github_cares_cares -D_GNU_SOURCE '-D_HAS_EXCEPTIONS=0' -DHAVE_CONFIG_H '--sysroot=external/androidndk/ndk/platforms/android-29/arch-x86_64' -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem external/androidndk/ndk/sources/android/support/include -isystemexternal/androidndk/ndk/sysroot/usr/include -c external/com_github_cares_cares/bitncmp.c -o bazel-out/android-x86_64-dbg/bin/external/com_github_cares_cares/_objs/ares/bitncmp.pic.o)
# Configuration: e530bf6bc4206cb6b15d1b7417fef783495fdc598b0acff74769e88b96f50e3a
# Execution platform: @local_config_platform//:host
ERROR: /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/external/boringssl/BUILD:161:11: Compiling src/crypto/cipher_extra/e_rc2.c failed: (Exit 1): linux-sandbox failed: error executing command
(cd /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/sandbox/linux-sandbox/501/execroot/myproj-client && \
exec env - \
PATH=/opt/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/home/oakad/.dotnet/tools:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/home/oakad/.cache/bazel/_bazel_oakad/install/929d880fb7a5aae3fbd9d0518b974383/linux-sandbox -t 15 -w /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/sandbox/linux-sandbox/501/execroot/myproj-client -w /tmp -w /dev/shm -D -- external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -gcc-toolchain external/androidndk/ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -target x86_64-none-linux-android -fPIC -isystemexternal/androidndk/ndk/sysroot/usr/include/x86_64-linux-android '-D__ANDROID_API__=29' -no-canonical-prefixes -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -funwind-tables -fstack-protector-strong -fno-addrsig '-Werror=return-type' '-Werror=int-to-pointer-cast' '-Werror=pointer-to-int-cast' '-Werror=implicit-function-declaration' -O0 -g -MD -MF bazel-out/android-x86_64-dbg/bin/external/boringssl/_objs/crypto/e_rc2.pic.d '-frandom-seed=bazel-out/android-x86_64-dbg/bin/external/boringssl/_objs/crypto/e_rc2.pic.o' -fPIC -iquote external/boringssl -iquote bazel-out/android-x86_64-dbg/bin/external/boringssl -isystem external/boringssl/src/include -isystem bazel-out/android-x86_64-dbg/bin/external/boringssl/src/include -DOPENSSL_NO_ASM '--sysroot=external/androidndk/ndk/platforms/android-29/arch-x86_64' -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include -isystem external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem external/androidndk/ndk/sources/android/support/include -isystemexternal/androidndk/ndk/sysroot/usr/include -c external/boringssl/src/crypto/cipher_extra/e_rc2.c -o bazel-out/android-x86_64-dbg/bin/external/boringssl/_objs/crypto/e_rc2.pic.o)
1666785042.500807052: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1666785042.500832910: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1666785042.501048092: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 39235
1666785042.501064925: src/main/tools/linux-sandbox-pid1.cc:650: Pid1Main started
1666785042.501123199: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1666785042.501206600: src/main/tools/linux-sandbox-pid1.cc:269: working dir: /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/sandbox/linux-sandbox/501/execroot/myproj-client
1666785042.501225700: src/main/tools/linux-sandbox-pid1.cc:301: writable: /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/sandbox/linux-sandbox/501/execroot/myproj-client
1666785042.501234827: src/main/tools/linux-sandbox-pid1.cc:301: writable: /tmp
1666785042.501243071: src/main/tools/linux-sandbox-pid1.cc:301: writable: /dev/shm
1666785042.501307251: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /
1666785042.501316552: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /dev
1666785042.501321447: src/main/tools/linux-sandbox-pid1.cc:371: remount rw: /dev/shm
1666785042.501325876: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /dev/pts
1666785042.501329584: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /dev/hugepages
1666785042.501333886: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /dev/mqueue
1666785042.501337899: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /proc
1666785042.501342723: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /proc/sys/fs/binfmt_misc
1666785042.501348797: src/main/tools/linux-sandbox-pid1.cc:391: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1666785042.501353713: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /proc/sys/fs/binfmt_misc
1666785042.501358158: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys
1666785042.501363333: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/firmware/efi/efivars
1666785042.501379050: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/kernel/security
1666785042.501393037: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/fs/cgroup
1666785042.501421276: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/fs/pstore
1666785042.501431031: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/fs/bpf
1666785042.501438296: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/kernel/debug
1666785042.501446224: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/kernel/tracing
1666785042.501454107: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/kernel/config
1666785042.501461507: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /sys/fs/fuse/connections
1666785042.501470335: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /run
1666785042.501478021: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /run/user/1000
1666785042.501485657: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /run/user/1000/doc
1666785042.501492999: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /run/user/1000/gvfs
1666785042.501499888: src/main/tools/linux-sandbox-pid1.cc:371: remount rw: /tmp
1666785042.501506582: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /boot
1666785042.501522202: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /boot1
1666785042.501528864: src/main/tools/linux-sandbox-pid1.cc:371: remount ro: /var/lib/docker/btrfs
1666785042.501535002: src/main/tools/linux-sandbox-pid1.cc:391: remount(nullptr, /var/lib/docker/btrfs, nullptr, 2101281, nullptr) failure (Permission denied) ignored
1666785042.501541739: src/main/tools/linux-sandbox-pid1.cc:371: remount rw: /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/sandbox/linux-sandbox/501/execroot/myproj-client
1666785042.501549716: src/main/tools/linux-sandbox-pid1.cc:371: remount rw: /home/oakad/.cache/bazel/_bazel_oakad/7eb352506b5dcd38b5d31a07fab48830/sandbox/linux-sandbox/501/execroot/myproj-client
1666785042.501566869: src/main/tools/linux-sandbox-pid1.cc:371: remount rw: /tmp
1666785042.501574952: src/main/tools/linux-sandbox-pid1.cc:371: remount rw: /dev/shm
1666785042.501637009: src/main/tools/linux-sandbox-pid1.cc:460: calling fork...
1666785042.501727828: src/main/tools/linux-sandbox-pid1.cc:490: child started with PID 2
src/main/tools/linux-sandbox-pid1.cc:487: "execvp(external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang, 0x562b445030c0)": No such file or directory
1666785042.502007821: src/main/tools/linux-sandbox-pid1.cc:507: wait returned pid=2, status=0x100
1666785042.502015295: src/main/tools/linux-sandbox-pid1.cc:525: child exited normally with code 1
1666785042.502174418: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //android/test-app:test-app failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.776s, Critical Path: 0.47s
INFO: 17 processes: 17 internal.
FAILED: Build did NOT complete successfully
Hello @oakad, can you provide an example repo to reproduce the same. Thanks!
For me, this stuff breaks on the simplest of repos, always:
https://github.com/oakad/bazel_issue_16558
bazel build --config android-emu :test
Thanks for the example, I can see the same issue. It does seem that it might have something to do with sandboxing, because adding --spawn_strategy=standalone
gives a different error about headers (missing dependency declarations for the following files included by 'test.cpp'
). Not totally sure what's going on, because I see the same error even as far back as Bazel 3.0.0. This all used to work fine, and I don't think anything has changed with how android_ndk_repository pulls in the ndk into the sandbox.
You could try https://github.com/bazelbuild/rules_android_ndk which is Starlark-based replacement for the native (i.e. implemented in Java) android_ndk_repository.
With Bazel 5.1, I get the error with NDK 21e but not with NDK 20:
bazel_issue_16558$ echo 5.1.0 > .bazelversion
bazel_issue_16558$ ANDROID_NDK_HOME=~/android-ndk-r21e bazelisk build --config=android-emu test
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:test (40 packages loaded, 2547 targets configured).
INFO: Found 1 target...
ERROR: /usr/local/google/home/ahumesky/bazel-workspaces/bazel_issue_16558/BUILD.bazel:7:11: Compiling test.cpp failed: (Exit 1): clang failed: error executing command external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -gcc-toolchain external/androidndk/ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -target x86_64-none-linux-android -fPIC ... (remaining 37 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox
src/main/tools/linux-sandbox-pid1.cc:478: "execvp(external/androidndk/ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang, 0x664240)": No such file or directory
Target //:test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.092s, Critical Path: 0.13s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
bazel_issue_16558$ ANDROID_NDK_HOME=~/android-ndk-r20 bazelisk clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
bazel_issue_16558$ ANDROID_NDK_HOME=~/android-ndk-r20 bazelisk build --config=android-emu test
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:test (40 packages loaded, 6407 targets configured).
INFO: Found 1 target...
Target //:test up-to-date:
bazel-bin/libtest.a
INFO: Elapsed time: 6.925s, Critical Path: 0.71s
INFO: 3 processes: 1 internal, 2 linux-sandbox.
INFO: Build completed successfully, 3 total actions
Is it possible you were using NDK 20 with Bazel 5.1?
ls android/ndk/ 21.4.7075529 25.1.8937393
I've got v25 for the new rules_android_ndk. Haven't changed this config in months (the v21 bit, v25 is obviously new).
I also found out that upgrading build-tools past v30 breaks the build in a different way. :-)
Not sure if of interest, but I'm using sdkmanager to fetch and install those.
Ok, I think I figured it out what's going on..... This line in one of the ndk templates:
gets expanded to:
filegroup(
name = "x86_64-clang9.0.8-libcpp-all_files",
srcs = glob(["ndk/toolchains/llvm/**"], allow_empty = True) + glob([
"ndk/platforms/android-29/arch-x86_64/**/*",
"ndk/sources/cxx-stl/llvm-libc++/include/**/*",
"ndk/sources/cxx-stl/llvm-libc++abi/include/**/*",
"ndk/sources/android/support/include/**/*",
"ndk/sysroot/**/*",
"ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/**/*",
], allow_empty = True) + [
":libcpp-x86_64-dynamic-runtime-libraries",
":libcpp-x86_64-static-runtime-libraries",
],
)
and the glob(["ndk/toolchains/llvm/**"])
should be picking up clang:
bazel_issue_16558/bazel-bazel_issue_16558/external/androidndk$ ls ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
But it returns nothing (confirmed through bazel query
). Glob does not traverse package boundaries... and sure enough:
~/android-ndk-r21e$ find . -type f | grep BUILD
./sources/third_party/googletest/test/BUILD.bazel
./toolchains/llvm/prebuilt/linux-x86_64/BUILD
./toolchains/llvm/prebuilt/linux-x86_64/sysroot/BUILD
There are a few BUILD files in the NDK, and right in the path that we want for clang, so glob() doesn't pick them up.
Removing those build files:
~/android-ndk-r21e$ mv ./toolchains/llvm/prebuilt/linux-x86_64/BUILD ./toolchains/llvm/prebuilt/linux-x86_64/BUILD.back
~/android-ndk-r21e$ mv ./toolchains/llvm/prebuilt/linux-x86_64/sysroot/BUILD ./toolchains/llvm/prebuilt/linux-x86_64/sysroot/BUILD.back
and it works now:
bazel_issue_16558$ ANDROID_NDK_HOME=~/android-ndk-r21e bazel build --config=android-emu test
INFO: Analyzed target //:test (39 packages loaded, 6615 targets configured).
INFO: Found 1 target...
Target //:test up-to-date:
bazel-bin/libtest.a
INFO: Elapsed time: 3.014s, Critical Path: 0.77s
INFO: 3 processes: 1 internal, 2 linux-sandbox.
INFO: Build completed successfully, 3 total actions
Where could those build files have come from? They're not in the NDK zip:
$ unzip -l android-ndk-r21e-linux-x86_64.zip | grep BUILD
13532 2021-01-11 13:34 android-ndk-r21e/sources/third_party/googletest/test/BUILD.bazel
What's in the build files anyway?
$ cat ./toolchains/llvm/prebuilt/linux-x86_64/BUILD.back
"""Declarations for the NDK's Clang directory."""
load("@rules_android//rules/ndk:ndk_cc_toolchain_config.bzl", "ndk_cc_toolchain_config_rule")
package(default_visibility = ["//visibility:public"])
TARGET_SYSTEM_NAMES = (
"arm-linux-androideabi",
"aarch64-linux-android",
"i686-linux-android",
"x86_64-linux-android",
)
....
Loading ndk_cc_toolchain_config.bzl
? Well, that's familiar:
https://github.com/bazelbuild/rules_android_ndk/blob/main/ndk_cc_toolchain_config.bzl
So now it makes sense: These are from when the Starlark version of android_ndk_repository
was inadvertently leaving BUILD files in the real NDK directory: https://github.com/bazelbuild/rules_android_ndk/issues/3
So, just delete those files:
~/android-ndk-r21e$ rm toolchains/llvm/prebuilt/linux-x86_64/BUILD toolchains/llvm/prebuilt/linux-x86_64/sysroot/BUILD
I also found out that upgrading build-tools past v30 breaks the build in a different way. :-)
Are you getting something about dx.jar being missing? That should be fixed in bazel 6.0. You can try these flags with bazel 5.x to work around it:
--define=android_incremental_dexing_tool=d8_dexbuilder
--define=android_dexmerger_tool=d8_dexmerger
--nouse_workers_with_dexbuilder
So this is down to the bug in rules_android_ndk
? Nice find and also explains a lot.
We do need both NDK rule sets to work in parallel in order to ensure smooth transition. For example, in my test project I have one branch configured the old way and another using the new approach.
https://github.com/bazelbuild/rules_android_ndk/issues/3 is fixed so it shouldn't be a problem going forward
Description of the bug:
Was not happening with older versions of Bazel. Does not happen on Windows (where there's no sandbox)
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Compile any NDK project on Linux. In my case, I'm using: SDK API Level 28 Build tools version 30.0.3 NDK API Level 29 NDK version 21.4.7075529
All this used to work with Bazel v5.
Which operating system are you running Bazel on?
Linux
What is the output of
bazel info release
?release 5.3.1
If
bazel info release
returnsdevelopment version
or(@non-git)
, tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD
?No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
external/androidndk/ndk/toolchains/llvm
subfolder doesn't appear to get mounted into sandbox for relevant targets.