Envoy FIPS v1.29.x x86_64 build failures

Description

The x86_64 build for release.server_only of FIPS Envoy (v1.29.X) seems to error out at the same stage, and looks to be related to https://github.com/envoyproxy/envoy/issues/31874 and https://github.com/envoyproxy/envoy/pull/32901

Setup Env

export BAZEL_BUILD_EXTRA_OPTIONS="--define boringssl=fips --local_cpu_resources=12 --build_tag_filters=-nofips --local_ram_resources=HOST_RAM*0.80 --jobs=12 --progress_report_interval=60"
export BAZEL_EXTRA_TEST_OPTIONS="--test_tag_filters=-nofips --test_env=ENVOY_IP_TEST_VERSIONS=v4only --local_ram_resources=HOST_RAM*1 --local_cpu_resources=12 --test_output=errors --test_summary=detailed"

Command:

ci/run_envoy_docker.sh ci/do_ci.sh release.server_only

Error logs

v1.29.1

ld.lld: error: cannot preempt symbol: OPENSSL_ia32cap_P
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(crypto.c.o)
>>> referenced by cpu-intel.c:287 (../crypto/cpu-intel.c:287)
>>>               cpu-intel.c.o:(OPENSSL_cpuid_setup) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:133 (../crypto/cpu-intel.c:133)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-14: ·[0;1;31merror: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:493: qzip] Error 1
make[1]: Leaving directory '/build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip.build_tmpdir/utils'
make: *** [Makefile:568: all-recursive] Error 1

v1.29.2

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-14: ·[0;1;31merror: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:493: qzip] Error 1
make[1]: Leaving directory '/build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy/bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip.build_tmpdir/utils'
make: *** [Makefile:568: all-recursive] Error 1
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/Configure.log

Target //distribution/binary:release failed to build
ERROR: /source/contrib/exe/BUILD:23:16 Linking contrib/exe/envoy-static failed: (Exit 2): bash failed: error executing command (from target //contrib/qat/compression/qatzip/compressor/source:qatzip) 
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  /bin/bash -c bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh)
# Configuration: 7e5fb22f0622a366157af3db174db77e4136215ff4d0bf9d71bc7eb0ac911359
# Execution platform: @local_config_platform//:host

v1.29.3

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-14: ·[0;1;31merror: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:493: qzip] Error 1
make[1]: Leaving directory '/build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy/bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip.build_tmpdir/utils'
make: *** [Makefile:568: all-recursive] Error 1
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/Configure.log

[12,845 / 14,338] Compiling src/codegen/compiler.cc; 15s processwrapper-sandbox ... (25 actions running)
Target //distribution/binary:release failed to build
ERROR: /source/contrib/exe/BUILD:31:16 Linking contrib/exe/envoy-static failed: (Exit 2): bash failed: error executing command (from target //contrib/qat/compression/qatzip/compressor/source:qatzip) 
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  /bin/bash -c bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh)
# Configuration: 7e5fb22f0622a366157af3db174db77e4136215ff4d0bf9d71bc7eb0ac911359
# Execution platform: @local_config_platform//:host

@Jayden-Lind https://github.com/envoyproxy/envoy/pull/32901 was added to smooth the path to updating llvm

im guessing that the issue is related to the llvm version that you are using

are you using the Envoy build container (ie ./ci/run_envoy_docker.sh) ? That would ensure you have the correct versions

Thanks @phlax, I am, this is the command I'm running ci/run_envoy_docker.sh ci/do_ci.sh release.server_only

ok - so not compiler version in that case

trying to understand why this issue is affecting your build and not our CI

we could potentially backport the fPIC flag - but i think it should not be necessary

we test this in our CI using the compile_time_options target rather than the release target

probably worth testing adding --config=compile-time-options as that should provide most of the same bazel flags

Thanks @phlax

Current setup

BAZEL_BUILD_EXTRA_OPTIONS: "--define boringssl=fips --verbose_failures --show_progress --local_ram_resources=HOST_RAM*0.90 --config=compile-time-options"

BAZEL_EXTRA_TEST_OPTIONS: "--test_tag_filters=-nofips --test_env=ENVOY_IP_TEST_VERSIONS=v4only --local_ram_resources=HOST_RAM*0.90"

./ci/run_envoy_docker.sh ./ci/do_ci.sh release.server_only

Error message I now get on v1.29.3

ERROR: /source/contrib/qat/compression/qatzip/compressor/source/BUILD:17:15: Foreign Cc - Configure: Building qatzip failed: (Exit 2): bash failed: error executing command (from target //contrib/qat/compression/qatzip/compressor/source:qatzip) 
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/6776/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  /bin/bash -c bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh)
# Configuration: cab6da5920fc21644abbc4db03292404922deb4fbc440b1843d98b468d0bcd4c
# Execution platform: @local_config_platform//:host
...
ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6776/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:133 (../crypto/cpu-intel.c:133)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6776/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6776/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6776/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

@Jayden-Lind reading through the related issues/PRs that have landed on main i see now that we need to exclude qatzip properly

it seems like this issue was closed https://github.com/envoyproxy/envoy/issues/31874 despite the fix not being backported

i think at least we need to backport https://github.com/envoyproxy/envoy/pull/31910 - most likely https://github.com/envoyproxy/envoy/pull/31898 also

would you be able to raise backports for these?

actually, nm re backport/s - i can do it

@Jayden-Lind if you are able to checkout the branch from https://github.com/envoyproxy/envoy/pull/33470 could you confirm whether this fixes?

Hey @phlax, now getting this

[20,859 / 31,137] Executing genrule @boringssl_fips//:build; 376s processwrapper-sandbox ... (71 actions, 70 running)
ERROR: /source/source/common/quic/BUILD:573:17: Compiling source/common/quic/send_buffer_monitor.cc failed: (Exit 1): clang-14 failed: error executing command (from target //source/common/quic:send_buffer_monitor_lib) 
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/8701/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
  /opt/llvm/bin/clang-14 -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-stdlib=libc++' -MD -MF bazel-out/k8-opt/bin/source/common/quic/_objs/send_buffer_monitor_lib/send_buffer_monitor.d '-frandom-seed=bazel-out/k8-opt/bin/source/common/quic/_objs/send_buffer_monitor_lib/send_buffer_monitor.o' -gsplit-dwarf -g -DFMT_HEADER_ONLY -DSPDLOG_FMT_EXTERNAL -DENVOY_ADMIN_FUNCTIONALITY -DENVOY_ENABLE_FULL_PROTOS -DENVOY_ENABLE_YAML -DENVOY_ENABLE_HTTP_DATAGRAMS -DENVOY_MOBILE_ENABLE_LISTENER -DENVOY_MOBILE_XDS -DBORINGSSL_FIPS '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/com_github_gabime_spdlog -iquote bazel-out/k8-opt/bin/external/com_github_gabime_spdlog -iquote external/com_github_fmtlib_fmt -iquote bazel-out/k8-opt/bin/external/com_github_fmtlib_fmt -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/utf8_range -iquote bazel-out/k8-opt/bin/external/utf8_range -iquote external/com_github_google_quiche -iquote bazel-out/k8-opt/bin/external/com_github_google_quiche -iquote external/boringssl_fips -iquote bazel-out/k8-opt/bin/external/boringssl_fips -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_nowkt -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf_lite -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_align -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/lite -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/port_def -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_allocation_policy -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_config -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/arena_cleanup -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/string_block -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/varint_shuffle -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/io_win32 -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/gzip_stream -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/stubs/_virtual_includes/stubs -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/printer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/zero_copy_sink -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/io/_virtual_includes/tokenizer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/any_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/empty_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/struct_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/wrappers_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/_virtual_includes/protobuf -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/protobuf -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/api_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/source_context_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/type_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/duration_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/field_mask_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/timestamp_proto -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/compiler/_virtual_includes/importer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/delimited_message_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/differencer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/field_mask_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/json_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/json -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/parser -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/descriptor_traits -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/lexer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/message_path -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/zero_copy_buffered_stream -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/untyped_message -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/type_resolver_util -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/_virtual_includes/descriptor_legacy -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/unparser -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/json/_virtual_includes/writer -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/util/_virtual_includes/time_util -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/logger_impl_lib_standard -isystem external/com_github_gabime_spdlog/include -isystem bazel-out/k8-opt/bin/external/com_github_gabime_spdlog/include -isystem external/com_github_fmtlib_fmt/include -isystem bazel-out/k8-opt/bin/external/com_github_fmtlib_fmt/include -isystem bazel-out/k8-opt/bin/external/envoy/bazel/foreign_cc/zlib/include -isystem external/boringssl_fips/boringssl/include -isystem bazel-out/k8-opt/bin/external/boringssl_fips/boringssl/include '-DABSL_MIN_LOG_LEVEL=4' -fdebug-types-section -fPIC -Wno-deprecated-declarations '-Wno-error=deprecated-enum-enum-conversion' '-std=c++17' '-std=c++20' -Wall -Wextra -Werror -Wnon-virtual-dtor -Woverloaded-virtual -Wold-style-cast -Wformat -Wformat-security -Wvla -Wno-deprecated-declarations -Wreturn-type -ggdb3 -fno-limit-debug-info -Wgnu-conditional-omitted-operand -Wc++2a-extensions -Wrange-loop-analysis -DGPERFTOOLS_TCMALLOC -DENVOY_OBJECT_TRACE_ON_DUMP -DENVOY_DISABLE_DEPRECATED_FEATURES -DENVOY_LOG_DEBUG_ASSERT_IN_RELEASE -DENVOY_ENABLE_UHV -DENVOY_STATIC_EXTENSION_REGISTRATION -DENVOY_NORMALIZE_PATH_BY_DEFAULT -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c source/common/quic/send_buffer_monitor.cc -o bazel-out/k8-opt/bin/source/common/quic/_objs/send_buffer_monitor_lib/send_buffer_monitor.o)
# Configuration: e5b684e418a7852cd3fb36b4ab7dbbb0d8891876f4f94a5983dd0693865afbaf
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from source/common/quic/send_buffer_monitor.cc:1:
./source/common/quic/send_buffer_monitor.h:3:10: ·[0;1;31mfatal error: 'quiche/quic/core/quic_stream.h' file not found
#include "quiche/quic/core/quic_stream.h"
·[0;1;32m         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[20,903 / 31,137] Executing genrule @boringssl_fips//:build; 377s processwrapper-sandbox ... (32 actions running)
[20,934 / 31,137] Executing genrule @boringssl_fips//:build; 378s processwrapper-sandbox
INFO: Elapsed time: 455.179s, Critical Path: 379.01s
INFO: 20935 processes: 12270 internal, 1 local, 8663 processwrapper-sandbox, 1 worker.
FAILED: Build did NOT complete successfully

i think thats progress - the lib in the error is tagged nofips so it should be ignored

in your original command/env you had --build_tag_filters=-nofips set - not 100% that this will resolve but worth checking

i reckon we do want to land that backport tho

Ahh you're right, I missed reverting my troubleshooting changes, and have added --build_tag_filters=-nofips back This works now 👍

Thanks for your help @phlax

@phlax, can see https://github.com/envoyproxy/envoy/pull/33470 this being merged, thanks again for getting this sorted 😄 any ideas on when this backfix will be available on the tagged releases for v1.29.X?

I've also discovered with the new v1.30.0 release, build errors as well with this backport cherry picked and without it.

Build Env

export BAZEL_BUILD_EXTRA_OPTIONS="--define boringssl=fips --local_cpu_resources=12 --build_tag_filters=-nofips --local_ram_resources=HOST_RAM*0.80 --jobs=12 --progress_report_interval=60"
export BAZEL_EXTRA_TEST_OPTIONS="--test_tag_filters=-nofips --test_env=ENVOY_IP_TEST_VERSIONS=v4only --local_ram_resources=HOST_RAM*1 --local_cpu_resources=12 --test_output=errors --test_summary=detailed"

ci/run_envoy_docker.sh ci/do_ci.sh release.server_only

Errors

ERROR: /source/contrib/cryptomb/private_key_providers/source/BUILD:17:12: Foreign Cc - CMake: Building ipp-crypto failed: (Exit 1): bash failed: error executing command (from target //contrib/cryptomb/private_key_providers/source:ipp-crypto)
rules_foreign_cc: Build failed!
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/6564/execroot/envoy && \
rules_foreign_cc: Keeping temp build directory and dependencies directory for debug.
  exec env - \
rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify --sandbox_debug Bazel command line flag.
    BAZEL_COMPILER=clang \
rules_foreign_cc: Printing build logs:
    BAZEL_CXXOPTS='-stdlib=libc++' \
_____ BEGIN BUILD LOGS _____
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \

    BAZEL_LINKOPTS=-lm:-pthread \
Bazel external C/C++ Rules. Building library ipp-crypto
    CC=clang \

    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
Environment:______________
    LDFLAGS='-stdlib=libc++' \
BUILD_SCRIPT=bazel-out/k8-opt/bin/contrib/cryptomb/private_key_providers/source/ipp-crypto_foreign_cc/build_script.sh
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
EXT_BUILD_ROOT=/build/bazel_root/base/sandbox/processwrapper-sandbox/6564/execroot/envoy
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
BUILD_LOG=bazel-out/k8-opt/bin/contrib/cryptomb/private_key_providers/source/ipp-crypto_foreign_cc/CMake.log
  /bin/bash -c bazel-out/k8-opt/bin/contrib/cryptomb/private_key_providers/source/ipp-crypto_foreign_cc/wrapper_build_script.sh)
PWD=/build/bazel_root/base/sandbox/processwrapper-sandbox/6564/execroot/envoy
# Configuration: 432c3219619c84b8f600616fafed665473a773f8c66e74e4684f8371d8304543
LLVM_CONFIG=/opt/llvm/bin/llvm-config
# Execution platform: @local_config_platform//:host
CXX=clang++

FAILED: src/CMakeFiles/crypto_mb_s.dir/common/ifma_cvt52.c.o 
/opt/llvm/bin/clang-14 -DSIMD_LEN=512 -DUSE_AMS_5x -I/build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/external/com_github_intel_ipp_crypto_crypto_mb/sources/ippcp/crypto_mb/include -I/usr/include/X11 -isystem /build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/bazel-out/k8-opt/bin/contrib/cryptomb/private_key_providers/source/ipp-crypto.ext_build_deps/include -falign-functions=32 -std=c99 -Wno-pointer-to-int-cast  -O3 -DNDEBUG -fvisibility=hidden  -march=icelake-server -mavx512dq -mavx512ifma -mavx512f -mavx512vbmi2 -mavx512cd -mavx512bw -mbmi2   -Wformat -Wformat-security -Werror=format-security -fcf-protection=full -D_FORTIFY_SOURCE=2 -fstack-protector -fpic -fPIC -Wall -Werror -MD -MT src/CMakeFiles/crypto_mb_s.dir/common/ifma_cvt52.c.o -MF src/CMakeFiles/crypto_mb_s.dir/common/ifma_cvt52.c.o.d -o src/CMakeFiles/crypto_mb_s.dir/common/ifma_cvt52.c.o -c /build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/external/com_github_intel_ipp_crypto_crypto_mb/sources/ippcp/crypto_mb/src/common/ifma_cvt52.c
/build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/external/com_github_intel_ipp_crypto_crypto_mb/sources/ippcp/crypto_mb/src/common/ifma_cvt52.c:196:10: error: implicit declaration of function 'BN_bn2lebinpad' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
         BN_bn2lebinpad(bn[i], d[i], byteLen);
         ^
/build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/external/com_github_intel_ipp_crypto_crypto_mb/sources/ippcp/crypto_mb/src/common/ifma_cvt52.c:196:10: note: did you mean 'BN_bn2binpad'?
/build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/bazel-out/k8-opt/bin/contrib/cryptomb/private_key_providers/source/ipp-crypto.ext_build_deps/include/openssl/bn.h:946:20: note: 'BN_bn2binpad' declared here
OPENSSL_EXPORT int BN_bn2binpad(const BIGNUM *in, uint8_t *out, int len);
                   ^
/build/bazel_root/base/sandbox/processwrapper-sandbox/5464/execroot/envoy/external/com_github_intel_ipp_crypto_crypto_mb/sources/ippcp/crypto_mb/src/common/ifma_cvt52.c:483:10: error: implicit declaration of function 'BN_bn2lebinpad' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
         BN_bn2lebinpad(bn[i], (unsigned char *)inp[i], byteLen);
         ^
2 errors generated.

I can raise a new issue for the v1.30.0 build issue

any ideas on when this backfix will be available on the tagged releases for v1.29.X?

there isnt a current plan to cut a release but im guessing a month or so - there is nothing stopping us cutting a release sooner - just needs a bit of work

I've also discovered with the new v1.30.0 release, build errors as well with this backport cherry picked and without it.

im a bit confused here - the backport was cherry-picked from main and 1.30 was cut from main so it should be on that branch already

I can raise a new issue for the v1.30.0 build issue

rather raise any issue against main as the issue must be present there also

Thanks @phlax,

im a bit confused here - the backport was cherry-picked from main and 1.30 was cut from main so it should be on that branch already Sorry you are correct, didn't realise it was so far back in the git log.

I can see someone else has raised https://github.com/envoyproxy/envoy/issues/33585. I'm also receiving these same errors on main branch as well as v1.30.0, will update that thread 👍

@Jayden-Lind contrary to what i said previously there is an imminent set of patch releases

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

envoyproxy / envoy