Envoy build fails on PowerPC (ppc64le) architecture

[aishwaryabk]

Hi, I am trying to build envoy on rhel:7.6 ppc64le. I have successfully built its dependencies viz. Bazel v0.25.0, Go-v1.12.5, Cmake - 3.16.0-rc3, AdoptOpenJDK11, Ninja v1.8.2, Boringssl. However it generates errors while building it.

ERRORS: An error occurred during the fetch of repository 'com_github_gperftools_gperftools' Repository 'com_github_gperftools_gperftools' used the following cache hits instead of downloading the corresponding file. Hash '18574813a062eee487bc1b761e8024a346075a7cb93da19607af362dc09565ef' for https://github.com/gperftools/gperftools/archive/fc00474ddc21fff618fc3f009b46590e241e425e.tar.gz

An error occurred during the fetch of repository 'com_github_nghttp2_nghttp2'

/source_root/envoy/bazel/repositories.bzl:692:5: error loading package '@com_github_grpc_grpc//': in /root/.cache/bazel/_bazel_root/075d06ed77119217da9daa09c3a46431/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /root/.cache/bazel/_bazel_root/075d06ed77119217da9daa09c3a46431/external/build_bazel_rules_apple/apple/ios.bzl: in /root/.cache/bazel/_bazel_root/075d06ed77119217da9daa09c3a46431/external/build_bazel_rules_apple/apple/internal/ios_rules.bzl: in /root/.cache/bazel/_bazel_root/075d06ed77119217da9daa09c3a46431/external/build_bazel_rules_apple/apple/internal/aspects/swift_static_framework_aspect.bzl: in /root/.cache/bazel/_bazel_root/075d06ed77119217da9daa09c3a46431/external/build_bazel_rules_swift/swift/swift.bzl: in /root/.cache/bazel/_bazel_root/075d06ed77119217da9daa09c3a46431/external/build_bazel_rules_swift/swift/internal/swift_library.bzl: Extension file 'rules/common_settings.bzl' has errors and referenced by '//external:grpc'

I was building envoy so as to build Istio proxy. Would like some help on understanding the cause for the same. I am running it on a High end VM with good connectivity.

[junr03]

I am not sure if people have tried building on PowerPC before, or if there are known issues. cc @lizan?

[lizan]

I haven't either. I think IBM folks did, @cmluciano?

[cmluciano]

Yep, I will take a look at our builds. I should note that I have not built this on Redhat before and we are only running our CI against libraries contained within Debian Buster.

[cmluciano]

@aishwaryabk On a fresh install with debian buster I see the following but I think it is an error with me not passing in the correct flag

ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qadd: .text+0xC): relocation R_PPC64_TOC16_DS out of range: 428344 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qadd: .text+0x80): relocation R_PPC64_TOC16_DS out of range: 428352 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qmul: .text+0x138): relocation R_PPC64_TOC16_DS out of range: 428344 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qdiv: .text+0x1C8): relocation R_PPC64_TOC16_DS out of range: 428344 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qdiv: .text+0x1DC): relocation R_PPC64_TOC16_DS out of range: 428360 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qdiv: .text+0x240): relocation R_PPC64_TOC16_DS out of range: 428368 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x38): relocation R_PPC64_TOC16_DS out of range: 428376 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x58): relocation R_PPC64_TOC16_DS out of range: 428384 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x230): relocation R_PPC64_TOC16_DS out of range: 428376 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x258): relocation R_PPC64_TOC16_DS out of range: 428384 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixdfdi: .text+0x52C): relocation R_PPC64_TOC16_DS out of range: 428392 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixdfdi: .text+0x53C): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixdfdi: .text+0x588): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x5E4): relocation R_PPC64_TOC16_DS out of range: 428392 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x5EC): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x634): relocation R_PPC64_TOC16_DS out of range: 428392 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x638): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixunsdfsi: .text+0x698): relocation R_PPC64_TOC16_DS out of range: 428408 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixunssfsi: .text+0x6F8): relocation R_PPC64_TOC16_DS out of range: 428416 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __floatditf: .text+0x760): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Can you use a more recent bazel version and see if you have the same error?

[aishwaryabk]

@aishwaryabk On a fresh install with debian buster I see the following but I think it is an error with me not passing in the correct flag

ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qadd: .text+0xC): relocation R_PPC64_TOC16_DS out of range: 428344 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qadd: .text+0x80): relocation R_PPC64_TOC16_DS out of range: 428352 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qmul: .text+0x138): relocation R_PPC64_TOC16_DS out of range: 428344 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qdiv: .text+0x1C8): relocation R_PPC64_TOC16_DS out of range: 428344 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qdiv: .text+0x1DC): relocation R_PPC64_TOC16_DS out of range: 428360 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ibm-ldouble.o):(function __gcc_qdiv: .text+0x240): relocation R_PPC64_TOC16_DS out of range: 428368 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x38): relocation R_PPC64_TOC16_DS out of range: 428376 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x58): relocation R_PPC64_TOC16_DS out of range: 428384 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x230): relocation R_PPC64_TOC16_DS out of range: 428376 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixtfdi: .text+0x258): relocation R_PPC64_TOC16_DS out of range: 428384 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixdfdi: .text+0x52C): relocation R_PPC64_TOC16_DS out of range: 428392 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixdfdi: .text+0x53C): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixdfdi: .text+0x588): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x5E4): relocation R_PPC64_TOC16_DS out of range: 428392 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x5EC): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x634): relocation R_PPC64_TOC16_DS out of range: 428392 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixsfdi: .text+0x638): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixunsdfsi: .text+0x698): relocation R_PPC64_TOC16_DS out of range: 428408 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __fixunssfsi: .text+0x6F8): relocation R_PPC64_TOC16_DS out of range: 428416 is not in [-32768, 32767]
ld.lld-8: error: /usr/bin/../lib/gcc/powerpc64le-linux-gnu/8/libgcc.a(ppc64-fp.o):(function __floatditf: .text+0x760): relocation R_PPC64_TOC16_DS out of range: 428400 is not in [-32768, 32767]
ld.lld-8: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Can you use a more recent bazel version and see if you have the same error?

yes, its gets built with a higher bazel version.

[aishwaryabk]

Thank you all. I shall close this issue.

[aishwaryabk]

Reopening this issue as although the envoy build passes, the tests fail. I am on rhel:7.6/ppc64le.

ERROR: /envoy/test/server/BUILD:331:1: C++ compilation of rule '//test/server:filter_chain_benchmark_test' failed (Exit 1) gcc failed: error executing command /opt/rh/devtoolset-7/root/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' ... (remaining 490 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox In file included from test/server/filter_chain_benchmark_test.cc:19:0: bazel-out/ppc-fastbuild/bin/external/com_github_google_benchmark/_virtual_includes/benchmark/benchmark/benchmark.h:1071:16: error: 'virtual void benchmark::Fixture::SetUp(benchmark::State&)' was hidden [-Werror=overloaded-virtual] virtual void SetUp(State& st) { SetUp(const_cast<const State&>(st)); } ^~~~~ test/server/filter_chain_benchmark_test.cc:152:8: error: by 'virtual void Envoy::Server::FilterChainBenchmarkFixture::SetUp(const benchmark::State&)' [-Werror=overloaded-virtual] void SetUp(const ::benchmark::State& state) override { ^~~~~ In file included from external/com_github_fmtlib_fmt/include/fmt/format.h:60:0, from bazel-out/ppc-fastbuild/bin/source/common/common/_virtual_includes/fmt_lib/common/common/fmt.h:4, from bazel-out/ppc-fastbuild/bin/source/common/common/_virtual_includes/minimal_logger_lib/common/common/logger.h:10, from bazel-out/ppc-fastbuild/bin/source/common/common/_virtual_includes/assert_lib/common/common/assert.h:5, from bazel-out/ppc-fastbuild/bin/include/envoy/stats/_virtual_includes/refcount_ptr_interface/envoy/stats/refcount_ptr.h:7, from bazel-out/ppc-fastbuild/bin/include/envoy/stats/_virtual_includes/stats_interface/envoy/stats/histogram.h:8, from bazel-out/ppc-fastbuild/bin/include/envoy/stats/_virtual_includes/stats_macros/envoy/stats/stats_macros.h:5, from bazel-out/ppc-fastbuild/bin/include/envoy/upstream/_virtual_includes/host_description_interface/envoy/upstream/host_description.h:8, from bazel-out/ppc-fastbuild/bin/include/envoy/network/_virtual_includes/filter_interface/envoy/network/filter.h:8, from bazel-out/ppc-fastbuild/bin/include/envoy/network/_virtual_includes/connection_interface/envoy/network/connection.h:11, from test/server/filter_chain_benchmark_test.cc:4: external/com_github_fmtlib_fmt/include/fmt/core.h: In member function 'void fmt::v5::internal::arg_map<Context>::push_back(fmt::v5::internal::value<Context>) [with Context = fmt::v5::basic_format_context<std::back_insert_iterator<fmt::v5::internal::basic_buffer<char> >, char>]': external/com_github_fmtlib_fmt/include/fmt/core.h:950:8: note: the ABI of passing aggregates with 16-byte alignment has changed in GCC 5 void push_back(value<Context> val) { ^~~~~~~~~

This command bazel test //test/... --host_javabase=@local_jdk//:jdk --copt "-DENVOY_IGNORE_GLIBCXX_USE_CXX11_ABI_ERROR=1" --cxxopt=-std=gnu++0x generates the above output. Is there any flag to suppress the warnings?

I am using gcc 7. I have also removed the flag "-Werror".

[aishwaryabk]

@cmluciano could you please take a look. Thank You.

[cmluciano]

Running a build now with gcc to see what I get, normally I run just the build and I've been using clang-8.

[lizan]

Tests aren't built with gcc on CI, and I don't expect it will always pass.

[cmluciano]

I tried to spin a build but it ended up timing out do to some issues with my bazel cache, I have one running now again. @aishwaryabk Are you locked in to using gcc or is clang a possibility?

[aishwaryabk]

I have tried using clang by setting DCMAKE_C_COMPILER=/usr/bin/clang and DCMAKE_CXX_COMPILER=/usr/bin/clang++. However now, the build also fails

ERROR: /root/.cache/bazel/_bazel_root/37e3aec351bcd85a6ea8b58e3592ef6e/external/com_google_protobuf/BUILD:106:1: C++ compilation of rule '@com_google_protobuf//:protobuf_lite' failed (Exit 1) gcc failed: error executing command (cd /root/.cache/bazel/_bazel_root/37e3aec351bcd85a6ea8b58e3592ef6e/sandbox/processwrapper-sandbox/12/execroot/envoy && \ exec env - \ PATH=/bin:/usr/bin:/usr/local/bin \ PWD=/proc/self/cwd \ /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.d '-frandom-seed=bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.o' -iquote external/com_google_protobuf -iquote bazel-out/host/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src -isystem bazel-out/host/bin/external/com_google_protobuf/src -g0 -g0 -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-write-strings -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_google_protobuf/src/google/protobuf/generated_message_table_driven_lite.cc -o bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/generated_message_table_driven_lite.o) Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox gcc: error trying to exec 'cc1plus': execvp: No such file or directory Target //source/exe:envoy-static failed to build INFO: Elapsed time: 4.569s, Critical Path: 0.35s INFO: 0 processes. FAILED: Build did NOT complete successfully

@cmluciano Please correct me if I am wrong.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.