Contrib doesnt build on Linux gcc (ICU)

[phlax]

When attempting to build contrib with gcc it throws ICU related errors:

/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fdebug-types-section -fPIC -Wno-deprecated-declarations -std=c++17 -fPIC -DU_CHARSET_IS_UTF8=1 -DU_USING_ICU_NAMESPACE=0 -DUCONFIG_ONLY_HTML_CONVERSION=1 -DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_BREAK_ITERATION=1 -DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 -DUCONFIG_NO_TRANSLITERATION=1 -DUCONFIG_NO_REGULAR_EXPRESSIONS=1 -W -Wall -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long   -fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm -fuse-ld=gold -l:libstdc++.a -Wl,--gc-sections   -o ../../bin/makeconv gencnvex.o genmbcs.o makeconv.o ucnvstat.o -L../../lib -licutu -L../../lib -licui18n -L../../lib -licuuc -L../../stubdata -licudata -lpthread -lm
                                                                            makeconv.o:makeconv.cpp:DW.ref.__gxx_personality_v0: error: undefined reference to '__gxx_personality_v0'                                                                         ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::mutex::lock(): error: undefined reference to 'std::__throw_system_error(int)'                                            ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::once_flag::_Prepare_execution::~_Prepare_execution(): error: undefined reference to 'std::__once_callable'               ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::once_flag::_Prepare_execution::~_Prepare_execution(): error: undefined reference to 'std::__once_call'                   ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function umtx_cleanup: error: undefined reference to 'std::condition_variable::~condition_variable()'                                  ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function umtx_init::{lambda()#2}::operator()() const: error: undefined reference to 'std::condition_variable::condition_variable()'    ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function icu_72::umtx_initImplPreInit(icu_72::UInitOnce&): error: undefined reference to 'std::condition_variable::wait(std::unique_lock<std::mutex>&)'                                                                                                                                                                  ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function icu_72::umtx_initImplPostInit(icu_72::UInitOnce&): error: undefined reference to 'std::condition_variable::notify_all()'      ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function void std::call_once<void (&)()>(std::once_flag&, void (&)()): error: undefined reference to '__once_proxy'                    ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function void std::call_once<void (&)()>(std::once_flag&, void (&)()): error: undefined reference to 'std::__throw_system_error(int)'  ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (&)()>(std::once_flag&, void (&)())::{lambda()#1}>(void (&)())::{lambda()#1}::operator()() const: error: undefined reference to 'std::__once_callable'                                                                               ../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (&)()>(std::once_flag&, void (&)())::{lambda()#1}>(
void (&)()): error: undefined reference to 'std::__once_callable'                                                                                                                 
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<void (&)()>(std::once_flag&, void (&)())::{lambda()#1}>(
void (&)()): error: undefined reference to 'std::__once_call'                                                                                                                     
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::unique_lock<std::mutex>::lock(): error: undefined reference to 'std::__throw_system_error(int)'                          
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::unique_lock<std::mutex>::lock(): error: undefined reference to 'std::__throw_system_error(int)'                          
collect2: error: ld returned 1 exit status                                                                                                                                        
make[2]: *** [Makefile:81: ../../bin/makeconv] Error 1                                                                                                                            make[2]: Leaving directory '/b/f/w/bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build.build_tmpdir/tools/makeconv'                                                     
make[1]: *** [Makefile:47: all-recursive] Error 2                                                                                                                                 
make[1]: Leaving directory '/b/f/w/bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build.build_tmpdir/tools'                                                              
make: *** [Makefile:153: all-recursive] Error 2  

[phlax]

related/testing PR #31806

[ravenblackx]

I think this is fixed by #31814

[phlax]

unfortunately not - its still broken on ICU build

[lukidzi]

I think I have a similar issue related to CEL upgrade when building on OSX, I've tried mac 11.7.10 and 12.7.1 AWS ami:

ami-0da052d2225304b8f | amzn-ec2-macos-12.7.1-20231220-195030 ami-06bf601fad285cc69 | amzn-ec2-macos-11.7.10-20230922-233531

Undefined symbols for architecture x86_64:
  "cel::interop_internal::ProtoStructValueToMessageWrapper(cel::Value const&)", referenced from:
      cel::interop_internal::ToLegacyValue(google::protobuf::Arena*, cel::Handle<cel::Value> const&, bool) in libinterop.a(interop.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //contrib/exe:envoy-static failed to build

Both didn't work for me. Last time I used macos-11.7.10 to build 1.28.0 and had no issues.

Apple clang version 13.0.0

[phlax]

@lukidzi to clarify this ticket has nothing to do with macos or CEL

there are various other reports elsewhere - and as i have suggested there you should open a ticket relevant to your issue

i dont use or have any access to macos so its not something im likely to follow up on

[phlax]

seems its failing on all release branches - altho not sure if different error - this is 1.26

   gcc   ...  /b/f/w/external/com_github_unicode_org_icu/icu4c/source/tools/makeconv/ucnvstat.c
/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=redacted -D__TIMESTAMP__=redacted -D__TIME__=redacted -DABSL_MIN_LOG_LEVEL=4 -fPIC -Wno-deprecated-declarations -std=c++17 -fPIC -DU_CHARSET_IS_UTF8=1 -DU_USING_ICU_NAMESPACE=0 -DUCONFIG_ONLY_HTML_CONVERSION=1 -DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_BREAK_ITERATION=1 -DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 -DUCONFIG_NO_TRANSLITERATION=1 -DUCONFIG_NO_REGULAR_EXPRESSIONS=1 -W -Wall -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long   -fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm -fuse-ld=gold -l:libstdc++.a -Wl,--gc-sections   -o ../../bin/makeconv gencnvex.o genmbcs.o makeconv.o ucnvstat.o -L../../lib -licutu -L../../lib -licui18n -L../../lib -licuuc -L../../stubdata -licudata -lpthread -lm  
makeconv.o:makeconv.cpp:DW.ref.__gxx_personality_v0: error: undefined reference to '__gxx_personality_v0'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::mutex::lock(): error: undefined reference to 'std::__throw_system_error(int)'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function umtx_cleanup: error: undefined reference to 'std::condition_variable::~condition_variable()'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function umtx_init::{lambda()#2}::operator()() const: error: undefined reference to 'std::condition_variable::condition_variable()'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function icu_72::umtx_initImplPreInit(icu_72::UInitOnce&): error: undefined reference to 'std::condition_variable::wait(std::unique_lock<std::mutex>&)'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function icu_72::umtx_initImplPostInit(icu_72::UInitOnce&): error: undefined reference to 'std::condition_variable::notify_all()'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::call_once<void (&)()>(std::once_flag&, void (&)())::{lambda()#2}::operator()() const: error: undefined reference to 'std::__once_callable'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function void std::call_once<void (&)()>(std::once_flag&, void (&)()): error: undefined reference to 'std::__once_callable'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function void std::call_once<void (&)()>(std::once_flag&, void (&)()): error: undefined reference to 'std::__once_call'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function void std::call_once<void (&)()>(std::once_flag&, void (&)()): error: undefined reference to '__once_proxy'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function void std::call_once<void (&)()>(std::once_flag&, void (&)()): error: undefined reference to 'std::__throw_system_error(int)'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::unique_lock<std::mutex>::lock(): error: undefined reference to 'std::__throw_system_error(int)'
../../lib/libicuuc.a(umutex.ao):umutex.cpp:function std::unique_lock<std::mutex>::lock(): error: undefined reference to 'std::__throw_system_error(int)'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:81: ../../bin/makeconv] Error 1
make[2]: Leaving directory '/b/f/w/bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build.build_tmpdir/tools/makeconv'
make[1]: *** [Makefile:47: all-recursive] Error 2
make[1]: Leaving directory '/b/f/w/bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build.build_tmpdir/tools'
make: *** [Makefile:153: all-recursive] Error 2
make: Leaving directory '/b/f/w/bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build.build_tmpdir'
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-fastbuild/bin/bazel/foreign_cc/unicode_icu_build_foreign_cc/Configure.log

[phlax]

cc @realtimetodie

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[krinkinmu]

@phlax I think I know what the issue here is. I was able to reproduce the issue with --config=docker-gcc and was able to move past this error (though to dicover another issue) when I added -host_action_env=CC=gcc and --host_action_env=CXX=g++. It looks like configure_make rule used to build ICU relies on the host environment configuration and if we don't provide CXX flag, it basically falls back to gcc as a linker in ICU which ultimately results in the error you see above.

At least --config=docker-gcc is fixable by changing .bazelrc, I'm not entirely sure how to fix it for builds witout explicit config provided (e.g. when toolchain is automatically detected and only g++/gcc is available), but I may try to look into it, if you're ok with assigning the issue to me?

For the context, I waas made aware of this issue when some folks internally in Microsoft hit it, so that's why I'm looking at this.

[krinkinmu]

/assign @krinkinmu

[krinkinmu]

I think I was correct that /usr/bin/gcc is ultimately used as CXX in the configure script, but I was not correct that providing host_action_env fixes the issue. It was a silly mistake on my part - the build order in Bazel is not deterministic, so when I discovered a new issue with GCC build I incorrectly assumed that the previous issue is addressed, alas it wasn't.

That being said, after digging through the Bazel support for CC tolchains and how can we separate CC and CXX, I realized that the failure in ICU build happens when we build tools (e.g. binaries that come together with ICU distribution). I don't think that we actually need those binary tools - we only need a libraries (and those tools aren't used in the build itself). In the end, I could fix the issue by just not building ICU tools.

After fixing ICU build, I hit another minor issue in the postgres plugin - easy to fix. And the last issue that I hit was with the cel library is overloaded-virtual warning (the same as https://github.com/envoyproxy/envoy/pull/31814, but it was re-introduced after the fix) - I now testing a fix for that issue. If cel fix works, I will try to create a PR for the cel repo and once it's merged, I prepare a PR for Envoy fix (which will involve bumping cel library version, to avoid mainting a patch on the Envoy side).

[krinkinmu]

Created https://github.com/google/cel-cpp/pull/1048 in cel-cpp, to fix that part of the gcc build.

[krinkinmu]

Hmm... looking at the cel-cpp repo, I don't see many PRs created by humans that were actually merged in... Is it a mirror only repository? Let me see if I can find some instructions on how to contribute.

[krinkinmu]

I can't see any other ways to suggest a change to the code, so it seems like this repo is mostly copied to Github from Google internal repository (that's what I assume this copybara-service does). I will wait until my tests with gcc Envoy build finish and if I will not hear anything on the PR, I will create a cel-cpp patch in Envoy for now.

[fredyw]

cc @kyessenov for the https://github.com/google/cel-cpp/pull/1048

[krinkinmu]

@fredyw thank you! I also reached out to cel-lang-discuss@googlegroups.com as well, but if @kyessenov can help that would be the best.

[krinkinmu]

With gcc I hit another issue in my tests, basically for whaterver reason LTO build takes a very long time (e.g., hours), done in single thread (though it might be just the first pass of lto in gcc that has to be done in single thread) and ultimately fails on my machine after eating too much memory and getting killed.

I will try without LTO now to see if it compiles in principle, and then will see if maybe updating gcc toolchain makes it better without causing additional issues. As it stands now, even if all the things breaking compilation with gcc are fixed, not being able to link Envoy on a reasonably powerful workstation is not great.

[phlax]

couple of questions

• do you hit similar issues building without contrib?
• are you using --config=gcc

re 2nd q - this is our config in bazelrc

  93 # Use gold linker for gcc compiler.
  94 build:gcc --linkopt=-fuse-ld=gold --host_linkopt=-fuse-ld=gold
  95 build:gcc --test_env=HEAPCHECK=
  96 build:gcc --action_env=BAZEL_COMPILER=gcc
  97 build:gcc --action_env=CC=gcc --action_env=CXX=g++

[krinkinmu]

couple of questions

• do you hit similar issues building without contrib?

Didn't try non-contrib build yet, will certainly try it next.

• are you using --config=gcc

I'm using --config=docker-gcc, but --config=gcc is what I was going to look into to try different version of gcc.

re 2nd q - this is our config in bazelrc

  93 # Use gold linker for gcc compiler.
  94 build:gcc --linkopt=-fuse-ld=gold --host_linkopt=-fuse-ld=gold
  95 build:gcc --test_env=HEAPCHECK=
  96 build:gcc --action_env=BAZEL_COMPILER=gcc
  97 build:gcc --action_env=CC=gcc --action_env=CXX=g++

These looks reasonable to me, but I feel that it's not everything. I think that lto is enabled on the build by default somehow, even fastbuild, mostly based on the fact that gcc spends tons of time running lto1-wpa. I see some toolchain configurations related to rba and I think that's what docker-gcc build is using, but I don't understand those configs enough yet.

[phlax]

seems as tho docker-gcc doesnt use gcc - that seems incorrect to me

[krinkinmu]

seems as tho docker-gcc doesnt use gcc - that seems incorrect to me

Do you mean that it does not use --config=gcc?

I also though that it's a bit weird, but comparing with clang configs it's basically the same - --config=docker-clang also does not use --config=clang. I've been using --config=docker-clang for my builds mostly and it has been working without issues for me so far.

[phlax]

Do you mean that it does not use --config=gcc?

yeah - i would imagine they should work ~the same - my guess is that this is an oversight, not sure tho

--config=docker-clang also does not use --config=clang

this one may be for slightly different reasons - most of the clang configs are the same (or were) - iirc i juggled some flags around so they could recently

[phlax]

my original reason for asking was due to the fact that it expressly sets the linker to use gold - not sure why but might be related to mem/timeout issues

[krinkinmu]

A small update, I tried using plain --config=gcc without docker (still using gcc version 11), it also gets terminated by OOM-killer after using a ton of memory.

My naive attempt to disable LTO via --copt=-fno-lto, --cxxopt=-fno-lto and --linkopt=-fno-lto didn't work out of the box either (linker got very confused when it for some reason saw object files with gcc internal representation instead of the machine code).

my original reason for asking was due to the fact that it expressly sets the linker to use gold - not sure why but might be related to mem/timeout issues

Yeah, I see where you're coming from. Thank you, @phlax, for the suggestion!

I still have a few ideas to try out, so the work is still in progress. If none of them works, I will try to build it on a much bigger machine to at least confirm that it's not some gcc bug and it just does not have enough resources on my workstation and will take it from there.

[krinkinmu]

Good news, cel-cpp folks fixed the issue on their side, so cel build failure should be addressed now.

[krinkinmu]

Another small update. I was not able to figure out the linker issue and building even on a very large machine results in OOM when using gcc (I tried versions 11 and 12). On the bright side, I now know that the issue is triggered by a single contrib extension - VPP VCL. When I disable VPP VCL extension, everything builds fine (modulo gcc converting some warnings to errors here and there).

I don't know what exactly in the VPP VCL extension trips gcc yet, but regardless of that, I would think that the gcc behavior in this case is more likely to be a gcc bug rather than VPP VCL bug. At the moment I'm looking for potential build configuration option changes in the extension to see if we can help gcc to build it without issues as a potential work around and will see if I can come up with a reproducer and send a bug report to the gcc folks.

[phlax]

cc @florincoras

[florincoras]

@krinkinmu do you happen to have some logs? Also what gcc version are you using? VCL (part of fd.io VPP) currently runs a CI job that builds VPP with gcc 9.4

[krinkinmu]

@florincoras sorry, I don't have any meangiful logs since the failure I've seen is that linker runs for a long time continiously accumulating residential memory and eventually gets killed by OOM reaper. So it's not like compilation or linking failed with an error, the problem is that linking is so resource hungry that even 256GiB of RAM is not enough.

As for the compiler versions, I tried two:

1. gcc-11 (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
2. gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0

Both experience the same failure mode. The version 11 is what the docker-gcc is currently using, so that's why I was experiemnting with it. And I tried version 12 just to see if maybe it's a known bug that got fixed in a later release of gcc and didn't get backported.

I would like to also point out that it's an LTO build. I'm not entirely sure where the LTO is configured (I don't seem to see any explicit options enabling LTO), but the OOM happens during one of the LTO passes in the linker. And building without LTO is broken, because even when I give bazel options to disable LTO, for unclear reason, some of the dependencies., including VPP VCL, are still built with LTO enabled.

And just to avoid some silly issues withing my environment, I cleaned up bazel cache (with --expunge) and tried from scratch multiple times with no difference.

[florincoras]

No worries, at least that gave me something to look at. My envoy build machine doesn't have gcc-11 but I just ran an independent vpp build with gcc-11 and everything seems to be working fine.

Regarding LTO, it is applied to only one component, i.e., vppinfra, that vcl depends on. Could you add a patch at the end of vpp_vcl.patch wherein you comment out this line?

[krinkinmu]

@florincoras Your suggestion helped to move past this linking issue. I now getting other errors that, I think, indicate a different issue (see below). These errors suggest that some symbols missing in the VPP extension, but I saw similar issues with other Envoy modules when compiled with gcc. In other cases they manifested when I used -c opt, while other configurations work fine. So I think it's not really a VPP specific issue anymore as much as some issue when compiling optimized binaries with gcc. I think it manifests for VPP because it by default builds optimized binaries, regardless of the bazel configuration AFAIU.

SUBCOMMAND: # //contrib/exe:envoy-static [action 'Linking contrib/exe/envoy-static', configuration: efdfc2f379a3cae84691392e64ad2267e6fe30be4337824d25da362874352e1b, execution platform: @local_config_platform//:
host]                                                                                                                                                                                                              
(cd /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/execroot/envoy && \                                                                                                                        
  exec env - \                                                                                                                                                                                                     
    BAZEL_COMPILER=gcc \                                                                                                                                                                                           
    BAZEL_LINKLIBS=-l%:libstdc++.a \                                                                                                                                                                               
    BAZEL_LINKOPTS=-lm \                                                                                                                                                                                           
    CC=gcc \                                                                                                                                                                                                       
    CXX=g++ \                                                                                                                                                                                                      
    PATH=/bin:/usr/bin:/usr/local/bin \                                                                                                                                                                            
    PWD=/proc/self/cwd \                                                                                                                                                                                           
  /usr/bin/gcc @bazel-out/k8-fastbuild/bin/contrib/exe/envoy-static-2.params)                                                                                                                                      
# Configuration: efdfc2f379a3cae84691392e64ad2267e6fe30be4337824d25da362874352e1b                                                                                                                                  
# Execution platform: @local_config_platform//:host                                                                                                                                                                
ERROR: /mount/data/ws/envoy/envoy/contrib/exe/BUILD:31:16: Linking contrib/exe/envoy-static failed: (Exit 1): gcc failed: error executing command (from target //contrib/exe:envoy-static) /usr/bin/gcc @bazel-out/
k8-fastbuild/bin/contrib/exe/envoy-static-2.params                                                                                                                                                                                                                                                                                                                                                                                    
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging                                                                                                       /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:594: error: undefined referen
ce to 'svm_msg_q_lock_and_alloc_msg_w_ring'                                                                                                                                                                        /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:597: error: undefined referen
ce to 'svm_msg_q_msg_data'                                                                                                                                                                                         /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:605: error: undefined referen
ce to 'svm_msg_q_add_and_unlock'                                                                                                                                                                                   /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:594: error: undefined referen
ce to 'svm_msg_q_lock_and_alloc_msg_w_ring'                                                                                                                                                                        /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:597: error: undefined referen
ce to 'svm_msg_q_msg_data'                                                                                                                                                                                         /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:605: error: undefined referen
ce to 'svm_msg_q_add_and_unlock'                                                                                                                                                                                   /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:594: error: undefined referen
ce to 'svm_msg_q_lock_and_alloc_msg_w_ring'                                                                                                                                                                        /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:597: error: undefined referen
ce to 'svm_msg_q_msg_data'                                                                                                                                                                                         /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:605: error: undefined referen
ce to 'svm_msg_q_add_and_unlock'                                                                                                                                                                                   /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:649: error: undefined referen
ce to 'svm_msg_q_alloc_msg_w_ring'                                                                                                                                                                                 /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:650: error: undefined referen
ce to 'svm_msg_q_msg_data'                                                                                                                                                                                         /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:653: error: undefined referen
ce to 'svm_msg_q_add_and_unlock'                                                                                                                                                                                   /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:648: error: undefined referen
ce to 'svm_msg_q_or_ring_wait_prod'                                                                                                                                                                                /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:594: error: undefined referen
ce to 'svm_msg_q_lock_and_alloc_msg_w_ring'                                                                                                                                                                        /mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:1171: error: undefined reference to 'svm_msg_q_sub'  
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:1178: error: undefined reference to 'svm_msg_q_free_msg'                                                                                                                                                                                                                
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:34: error: undefined reference to 'svm_msg_q_sub_raw_batch'                                                                                                                                                                                                             
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2567: error: undefined reference to 'svm_msg_q_free_msg'                                                                                                                                                                                                                
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:34: error: undefined reference to 'svm_msg_q_sub_raw_batch'                                                                                                                                                                                                             
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:1227: error: undefined reference to 'svm_msg_q_free_msg'                                                                                                                                                                                                                
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:789: error: undefined reference to 'svm_fifo_dequeue'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2075: error: undefined reference to 'svm_ms[135/2198]
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:754: error: undefined referen
ce to 'svm_fifo_peek'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:762: error: undefined referen
ce to 'svm_fifo_peek'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:764: error: undefined referen
ce to 'svm_fifo_peek'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:769: error: undefined referen
ce to 'svm_fifo_dequeue_drop'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2075: error: undefined reference to 'svm_msg_q_wait'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:649: error: undefined referen
ce to 'svm_msg_q_alloc_msg_w_ring'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:648: error: undefined referen
ce to 'svm_msg_q_or_ring_wait_prod'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:787: error: undefined referen
ce to 'svm_fifo_peek'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:1407: error: undefined reference to 'fifo_segment_mai
n_init'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:1803: error: undefined reference to 'svm_msg_q_wait'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2192: error: undefined reference to 'svm_fifo_segment
s'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2187: error: undefined reference to 'svm_msg_q_wait'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2229: error: undefined reference to 'svm_fifo_dequeue
_drop'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:683: error: undefined referen
ce to 'svm_fifo_enqueue_segments'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:710: error: undefined referen
ce to 'svm_fifo_enqueue'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:649: error: undefined referen
ce to 'svm_msg_q_alloc_msg_w_ring'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:648: error: undefined referen
ce to 'svm_msg_q_or_ring_wait_prod'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:683: error: undefined referen
ce to 'svm_fifo_enqueue_segments'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:710: error: undefined referen
ce to 'svm_fifo_enqueue'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:649: error: undefined referen
ce to 'svm_msg_q_alloc_msg_w_ring'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:648: error: undefined referen
ce to 'svm_msg_q_or_ring_wait_prod'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:710: error: undefined referen
ce to 'svm_fifo_enqueue'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vnet/session/application_interface.h:683: error: undefined referen
ce to 'svm_fifo_enqueue_segments'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:4392: error: undefined reference to 'svm_msg_q_sub'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:4396: error: undefined reference to 'svm_msg_q_free_m
sg'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:34: error: undefined reference to 'svm_msg_q_sub_raw_
batch'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:34: error: undefined reference to 'svm_msg_q_sub_raw_
batch'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:3325: error: undefined reference to 'svm_msg_q_timedw
ait'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vppcom.c:2555: error: undefined reference to 'svm_msg_q_timedw
ait'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:404: error: undefined reference to 'vl_msg_api_allo
c'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:415: error: undefined reference to 'vl_msg_api_send
_shmem'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api.h:968: error: undefined reference to 'vl_api_string_len'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/sessi[74/2198]
n.api.h:332: error: undefined reference to 'vl_api_string_len'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api.h:333: error: undefined reference to 'vl_api_format_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api.h:920: error: undefined reference to 'vl_api_string_len'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api.h:229: error: undefined reference to 'vl_api_string_len'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api.h:230: error: undefined reference to 'vl_api_format_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api_fromjson.h:295: error: undefined reference to 'vl_api_c_string_to_api_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/bazel-out/k8-opt-exec-2B5CBBC6/bin/contrib/vcl/source/build.build_tmpdir/CMakeFiles/vnet/session/sessio
n.api_fromjson.h:112: error: undefined reference to 'vl_api_c_string_to_api_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:297: error: undefined reference to 'vl_client_get_f
irst_plugin_msg_id'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:317: error: undefined reference to 'vl_msg_api_conf
ig'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:317: error: undefined reference to 'vl_msg_api_conf
ig'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:317: error: undefined reference to 'vl_msg_api_conf
ig'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:317: error: undefined reference to 'vl_msg_api_conf
ig'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vlibapi/api_common.h:398: error: undefined reference to 'my_api_ma
in'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:503: error: undefined reference to 'socket_client_m
ain'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:505: error: undefined reference to 'vl_client_api_u
nmap'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vlibmemory/memory_client.h:77: error: undefined reference to 'my_m
emory_client_main'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:532: error: undefined reference to 'vl_socket_clien
t_connect2'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:542: error: undefined reference to 'vl_socket_clien
t_init_shm2'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:126: error: undefined reference to 'vl_api_from_api
_to_new_c_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:84: error: undefined reference to 'vl_socket_client
_recv_fd_msg2'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:96: error: undefined reference to 'vl_api_from_api_
to_new_c_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:110: error: undefined reference to 'svm_msg_q_set_e
ventfd'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:220: error: undefined reference to 'vl_api_from_api
_to_new_c_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:185: error: undefined reference to 'vl_socket_clien
t_recv_fd_msg2'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:197: error: undefined reference to 'vl_api_from_api
_to_new_c_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:211: error: undefined reference to 'svm_msg_q_set_e
ventfd'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:351: error: undefined reference to 'vl_msg_api_allo
c'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:381: error: undefined reference to 'vl_msg_api_send
_shmem'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:378: error: undefined reference to 'vl_api_vec_to_a
pi_string'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:389: error: undefined reference to 'vl_msg_api_allo
c'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:395: error: undefined reference to 'vl_msg_api_send
_shmem'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:570: error: undefined reference to 'vl_soc[12/2198]
t_disconnect2'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:572: error: undefined reference to 'vl_client_disco
nnect_from_vlib'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:329: error: undefined reference to 'vl_msg_api_allo
c'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:336: error: undefined reference to 'vl_msg_api_send
_shmem'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:720: error: undefined reference to 'vl_client_send_
disconnect'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_bapi.c:728: error: undefined reference to 'vl_socket_clien
t_recv_fd_msg2'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_cfg.c:298: error: undefined reference to 'vl_set_memory_ui
d'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_cfg.c:303: error: undefined reference to 'vl_set_memory_gi
d'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:436: error: undefined reference to 'fifo_segment
_attach'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:477: error: undefined reference to 'fifo_segment
_get_segment'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:478: error: undefined reference to 'fifo_segment
_delete'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:543: error: undefined reference to 'fifo_segment
_get_segment'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:544: error: undefined reference to 'fifo_segment
_alloc_fifo_w_offset'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:545: error: undefined reference to 'fifo_segment
_alloc_fifo_w_offset'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:551: error: undefined reference to 'fifo_segment
_get_segment'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:552: error: undefined reference to 'fifo_segment
_msg_q_attach'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:585: error: undefined reference to 'fifo_segment
_get_segment_if_valid'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:590: error: undefined reference to 'fifo_segment
_free_client_fifo'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:591: error: undefined reference to 'fifo_segment
_free_client_fifo'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:594: error: undefined reference to 'fifo_segment
_get_segment_if_valid'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:599: error: undefined reference to 'fifo_segment
_free_client_fifo'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:600: error: undefined reference to 'fifo_segment
_free_client_fifo'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:623: error: undefined reference to 'fifo_segment
_get_segment'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:624: error: undefined reference to 'fifo_segment
_msg_q_attach'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:647: error: undefined reference to 'fifo_segment
_msg_qs_discover'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:672: error: undefined reference to 'fifo_segment
_alloc_chunk_w_slice'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:673: error: undefined reference to 'fifo_segment
_chunk_offset'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:689: error: undefined reference to 'fifo_segment
_duplicate_fifo'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:690: error: undefined reference to 'fifo_segment
_duplicate_fifo'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:694: error: undefined reference to 'svm_fifo_add
_subscriber'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_private.c:695: error: undefined reference to 'svm_fifo_add
_subscriber'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_sapi.c:93: error: undefined reference to 'svm_msg_q_set_ev
entfd'
/mount/data/bazel/cache/_bazel_kmu/ca4c8634fe188ae2080ff00715ba1313/sandbox/linux-sandbox/35/execroot/envoy/external/com_github_fdio_vpp_vcl/src/vcl/vcl_sapi.c:246: error: undefined reference to 'svm_msg_q_set_e
ventfd'
collect2: error: ld returned 1 exit status                                                               
Target //contrib/exe:envoy-static failed to build                                                        
Use --verbose_failures to see the command lines of failed build steps.                                                                                                                                             
INFO: Elapsed time: 3429.496s, Critical Path: 1579.28s                                                   
INFO: 12412 processes: 3157 internal, 9253 linux-sandbox, 1 local, 1 worker.                                                                                                                                       
FAILED: Build did NOT complete successfully

[krinkinmu]

I would like to change the mode of operation on this bug a little bit. Given that there were a bunch of issues with gcc build that are unrelated to each other, I would like to prepare a few small PRs for issues that I know how to fix and then return to the linking issues with optimized gcc builds to close this bug.

[phlax]

@krinkinmu sounds like a great idea

[florincoras]

Regarding the missing symbols, if you want to try out debug vcl binaries, try changing this to Debug

[krinkinmu]

The summary of the current state:

1. I can build with GCC (using both --config=gcc and --config=docker-gcc) after disabling a few warnings that are normally turned into errors
2. When using -c opt builds fail at the linking stage with missing symbols

Warnings don't seem like a big deal - we can always make those non-fatal at least. The linking issue is more intersting and I will try to resolve that one first.

[krinkinmu]

I took one of the functions referenced in the linker error and tried to see where it's coming from and where it should defined to get some anchors in the code to start from:

bazel-out/k8-opt/bin/external/v8/_objs/v8_libshared_noicu/macro-assembler-shared-ia32-x64.o(.debug_addr+0x180): error: undefined reference to 'v8::internal::Assembler::vpsllw(v8::internal::XMMRegister, v8::inter
nal::XMMRegister, v8::internal::XMMRegister)'

I could actually find a reference to this function in the code: https://github.com/v8/v8/blob/bb6560830541d9d937e0ae2e2164d8708362fcae/src/codegen/shared-ia32-x64/macro-assembler-shared-ia32-x64.cc#L473.

What I could not find though is where the function is implemented. I know that the version of instruction with 3 registers exists in Intel specs and I could find in V8 code the version of this instruction with 2 registers and an immediate. However, I could not find any place that defines a version with 3 registers.

It's not super easy to search through V8 code generation code because it's somewhat macro heavy, so to double check I did the following:

cat bazel-out/k8-opt/bin/contrib/exe/envoy-static-2.params | grep 'bazel-out/k8-opt/.*[.]o' | xargs nm -C 2>/dev/null | grep -B10 -A10 'vpsllw'

that also didn't yield the right symbol in any of the object files that were referenced in the linker command (it did find a version of vpsllw with 2 registers and an immediate, but not the version with 3 registers - the same thing I had after looking through the V8 code).

So unless I made a mistake somewhere, the linking error might actually be legit in the sense that this function is referenced and does not exist. Because the issue only manifests when building with -c opt it looks like one of the following has to be true:

1. Non-optimized build somehow does define the right version of vpsllw
2. Non-optimized build does not actually need a version of vpsllw - somehow fastbuild manages to optimzie the thing away, while the optimized build does not
3. Bazel just does not include all the files that are needed in the final linker command line

I'm going to repeat the same thing I did above for fastbuild to check if somehow maybe options 1 or 2 are true with nm tool next. And if it turns out that the right version of vpsllw acturally exists in the non-optimized code, then I will at least know where it's defined and can check if the object file exists in optimized builds.

[krinkinmu]

In unoptimized build there is indeed no reference to vpsllw with 3 registers in any of the object files (and I checked that the object file that refers to it in optimized builds is also there in unoptimized builds, so the refernce should be there if it wasn't cut off somehow).

If it was just a single undefined reference in v8 I would consider it a typo or a bug that just went unnoticed for a while. However, the number of undefined references coming from the v8 code is rather significant, so it's hard to believe that it's just a one off typo or mistake.

[krinkinmu]

Oh, I just spotted a thing that I think is important. Just like error in https://github.com/envoyproxy/envoy/issues/31807#issuecomment-2469194004 all other errors are also referring to the debug_addr section.

This is important because this section does not contain an actually used code. debug_addr section contains debugging information that requires linker resolution (e.g. when we need linker to fill in an address of a function or something after linking). So I think that what we have here is a failure to handle debugging information in the optimize binary and not an actual missing function definition (e.g., I still think that in V8 case the right version of vpsllw is not defined, but it does not matter if it's not actually used in the final binary).

[krinkinmu]

Compiling v8 without debugging information with -c opt does remove the undefined references in v8 code, so this confirms that the problem is limited to debuggin info (or at least to inconsistency between the debugging info and the actual emitted code).

The problem also looks very similar to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81993 that was reported and fixed in gcc quite a while ago (long before the gcc 11 or gcc 12, that I'm using now, were released).

And after some more searching I found https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110885. And from that report it looks like the following things are of interest:

1. The issue is triggered by a combination of -gsplit-dwarf (this is for debug fission) and -fdebug-types-section (this is to facilitate removing duplicate debug info entries by linker, so it's an optimization of a sort)
2. It seems like it could be mitigated by one of the following: a. disabling fission completely on gdb (I don't like it, since it diverges clang and gcc even more) b. dropping fdebug-types-section and losing this optimization on gcc c. using -gdwarf-3 - I don't know exactly how it mitigates the issue, but maybe it just renders fdebug-types-section mute by restricting gcc to use version of dwarf that does not actually benefit from fdebug-types-section?
3. It looks like fission support and fdebug-types-section are not maintained in gcc, which raises a question whether we should keep using those with gcc going forward.

For now I'm leaning towards just finding a way to disable fdebug-types-section. It would preserve fission with a potential negative effect on the size of debugging information, which I assume, is the most tolerable of downsides we can have.

[phlax]

It looks like fission support and fdebug-types-section are not maintained in gcc, which raises a question whether we should keep using those with gcc going forward.

hmm - yeah, i was wondering similar after reading the tickets you posted

... leaning towards just finding a way to disable fdebug-types-section. It would preserve fission with a potential negative effect on the size of debugging information, which I assume, is the most tolerable of downsides we can have.

yeah - i think that is probably ok - we only publish the clang-built debug binary - so it wouldnt affect us directly, and would be better than the current situation