search

bazelbuild / continuous-integration

Bazel's Continuous Integration Setup
https://buildkite.com
Apache License 2.0
259 stars 135 forks source link

Musl support in CI #1190

Open UebelAndre opened 3 years ago

UebelAndre commented 3 years ago

Are there any available hosts that support musl? If so, how would I go about configuring a cc_toolchain for it, otherwise, I suppose this would be a feature request to enable CI tests for musl (related to https://github.com/bazelbuild/rules_rust/pull/829).

philwo commented 3 years ago

@UebelAndre I never used musl - what would you need? Just apt install musl-dev in the Ubuntu 20.04 container or something more elaborate? 😀

UebelAndre commented 3 years ago

I think gcc-multi and musl-dev? But the hard part would probably be configuring the cc_toolchain or at least in a way that didn't require users to have intimate knowledge of the base image. Perhaps an alpine image (which is musl by default) would do the trick?

UebelAndre commented 3 years ago

Hi, any updates here? Is it possible to get an alpine image for use in CI?

philwo commented 3 years ago

@UebelAndre I'm not very experienced with Alpine Linux yet. I just tried this:

$ docker run --rm -it alpine:3.14
# mkdir test && cd test && touch WORKSPACE BUILD
# wget https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-linux-x86_64
# chmod +x bazel-4.2.1-linux-x86_64
# ./bazel-4.2.1-linux-x86_64 info
/bin/sh: ./bazel-4.2.1-linux-x86_64: not found

Running ldd bazel-4.2.1-linux-x86_64 shows that a lot of symbols cannot be resolved, probably because we link against glibc:

# ldd bazel-4.2.1-linux-x86_64
    /lib64/ld-linux-x86-64.so.2 (0x7f8d35ca6000)
    librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f8d35ca6000)
    libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f8d35ca6000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f8d35ca6000)
    libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f8d35ca6000)
Error loading shared library libstdc++.so.6: No such file or directory (needed by bazel-4.2.1-linux-x86_64)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by bazel-4.2.1-linux-x86_64)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f8d35ca6000)
Error relocating bazel-4.2.1-linux-x86_64: _ZdlPv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _Znwm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSsC1ERKSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs4swapERSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs4_Rep10_M_disposeERKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _Unwind_Resume: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSsC1EPKcRKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt18basic_stringstreamIcSt11char_traitsIcESaIcEED1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt17__throw_bad_allocv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt6localeD1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt25__throw_bad_function_callv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt8ios_baseD2Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs4_Rep10_M_destroyERKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs4_Rep9_S_createEmmRKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_begin_catch: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_rethrow: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_end_catch: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs7reserveEm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6appendEPKcm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6appendERKSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSsC1EPKcmRKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6assignERKSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt19__throw_logic_errorPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6insertEmPKcm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs7compareERKSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt18_Rb_tree_decrementPSt18_Rb_tree_node_base: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt20__throw_system_errori: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt20__throw_length_errorPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt18basic_stringstreamIcSt11char_traitsIcESaIcEEC1ESt13_Ios_Openmode: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSt15basic_stringbufIcSt11char_traitsIcESaIcEE3strEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6insertEmPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt18_Rb_tree_incrementPKSt18_Rb_tree_node_base: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs4findEcm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSsC1ERKSsmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_guard_acquire: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_guard_release: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_guard_abort: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt20__throw_out_of_rangePKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSt8__detail20_Prime_rehash_policy14_M_need_rehashEmmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt11_Hash_bytesPKvmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt8ios_base4InitC1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt8ios_base4InitD1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt6chrono3_V212system_clock3nowEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs7compareEmmRKSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt11this_thread11__sleep_forENSt6chrono8durationIlSt5ratioILl1ELl1EEEENS1_IlS2_ILl1ELl1000000000EEEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSdD2Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6assignEPKcm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt6thread15_M_start_threadESt10shared_ptrINS_10_Impl_baseEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt6thread4joinEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt9terminatev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs7compareEPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSt8__detail20_Prime_rehash_policy11_M_next_bktEm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_pure_virtual: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __gxx_personality_v0: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs12_M_leak_hardEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs4findEPKcmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt8ios_baseC2Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt9basic_iosIcSt11char_traitsIcEE4initEPSt15basic_streambufIcS1_E: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt6localeC1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSolsEi: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs7compareEmmPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6assignEPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs12_S_constructEmcRKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6resizeEmc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt13basic_filebufIcSt11char_traitsIcEEC1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt13basic_filebufIcSt11char_traitsIcEE4openEPKcSt13_Ios_Openmode: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt9basic_iosIcSt11char_traitsIcEE5clearESt12_Ios_Iostate: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSirsERi: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt13basic_filebufIcSt11char_traitsIcEE5closeEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt12__basic_fileIcED1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt13basic_filebufIcSt11char_traitsIcEED1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt14basic_ifstreamIcSt11char_traitsIcEED1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __realpath_chk: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSo5flushEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSolsEPSt15basic_streambufIcSt11char_traitsIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSo3putEc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSt5ctypeIcE13_M_widen_initEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt16__throw_bad_castv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt8__detail15_List_node_base7_M_hookEPS0_: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_allocate_exception: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt12bad_weak_ptrD1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_throw: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __memcpy_chk: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __fdelt_chk: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __strdup: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSsC1EmcRKSaIcE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt8__detail15_List_node_base9_M_unhookEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs9_M_mutateEmmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _Znam: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZdaPv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __strncpy_chk: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt9exceptionD2Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __longjmp_chk: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt6thread20hardware_concurrencyEv: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt12out_of_rangeD1Ev: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __cxa_free_exception: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSt12out_of_rangeC1ERKSs: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: __once_proxy: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZnwmRKSt9nothrow_t: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZdlPvRKSt9nothrow_t: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6appendEPKc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs5rfindEcm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6appendEmc: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs7replaceEmmPKcm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs6appendERKSsmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt18_Rb_tree_decrementPKSt18_Rb_tree_node_base: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNKSs12find_last_ofEPKcmm: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSo9_M_insertImEERSoT_: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSo9_M_insertIlEERSoT_: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt15__once_callable: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt11__once_call: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt9basic_iosIcSt11char_traitsIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt15basic_stringbufIcSt11char_traitsIcESaIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTISt12bad_weak_ptr: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVN10__cxxabiv121__vmi_class_type_infoE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt4cerr: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZSt7nothrow: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZNSs4_Rep20_S_empty_rep_storageE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTTSt19basic_ostringstreamIcSt11char_traitsIcESaIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTISt9exception: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt14basic_ifstreamIcSt11char_traitsIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt13basic_filebufIcSt11char_traitsIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt11logic_error: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt12out_of_range: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt18basic_stringstreamIcSt11char_traitsIcESaIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTTSt18basic_stringstreamIcSt11char_traitsIcESaIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt19basic_ostringstreamIcSt11char_traitsIcESaIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTTSt14basic_ifstreamIcSt11char_traitsIcEE: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt12bad_weak_ptr: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTISt12out_of_range: symbol not found
Error relocating bazel-4.2.1-linux-x86_64: _ZTVSt15basic_streambufIcSt11char_traitsIcEE: symbol not found

So it seems like this might not work. If you have an idea how we could make Bazel work under Alpine, I'd be happy to try it!

If it helps you, I'm tracking a feature request that allows projects on Bazel CI to install arbitrary packages using sudo apt install: https://github.com/bazelbuild/continuous-integration/issues/1214 - that way you could at least install musl-dev and gcc-multi. Maybe @hlopko could help us figure out how to create a cc_toolchain to use it?

UebelAndre commented 3 years ago

I don't think you can use any of the released binaries from 4.2.1 since they were likely all compiled on glibc system. The solution will most likely involve bootstrapping or finding some resolution to https://github.com/bazelbuild/bazel/issues/5891

I attempted the bootstrapping myself but am out of my element here

docker run --rm -it alpine:3.14
# Installing deps
apk add python3 bash openjdk11 openjdk8 gcc g++ musl-dev zip linux-headers
# Downloading Bazel
wget https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-dist.zip
# Extracting it
mkdir bazel && unzip bazel-4.2.1-dist.zip -d bazel && cd bazel
# Bootstrapping
env EXTRA_BAZEL_ARGS="--host_javabase=@bazel_tools//tools/jdk:remote_jdk11 \
--javabase=@bazel_tools//tools/jdk:remote_jdk11 \
--host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java11 \
--java_toolchain=@bazel_tools//tools/jdk:toolchain_java11" bash ./compile.sh

This gets me as far as this error for which I have no solution or context for (been years since I worked with Java and never bootstrapped Bazel before).

ERROR: /tmp/bazel_XXEgBhoH/out/external/bazel_tools/tools/jdk/BUILD:346:14: Action external/bazel_tools/tools/jdk/platformclasspath_classes/DumpPlatformClassPath.class failed: (Exit -1): javac failed: error executing command
  (cd /tmp/bazel_XXEgBhoH/out/execroot/io_bazel && \
  exec env - \
  external/remotejdk11_linux/bin/javac -source 8 -target 8 -Xlint:-options -cp external/remotejdk11_linux/lib/tools.jar -d bazel-out/k8-opt/bin/external/bazel_tools/tools/jdk/platformclasspath_classes external/bazel_tools/tools/jdk/DumpPlatformClassPath.java)
Execution platform: //:default_host_platform
Action failed to execute: java.io.IOException: Cannot run program "/tmp/bazel_XXEgBhoH/out/execroot/io_bazel/external/remotejdk11_linux/bin/javac" (in directory "/tmp/bazel_XXEgBhoH/out/execroot/io_bazel"): error=2, No such file or directory
Target //src:bazel_nojdk failed to build

Does this shine light any a potential missing flag or dependency?

UebelAndre commented 2 years ago

Friendly ping here.