`dyld symbol not found` when Linking Tests Dynamically on MacOS

[chenhao94]

It happens on all the tests that are using config, the error message is:

dyld[2104]: symbol not found in flat namespace '__ZN8simdjson8internal11error_codesE'

This symbol was defined in external/simdjson/src/internal/error_tables.cpp

$ nm -C bazel-bin/external/simdjson/libsimdjson/lib/libsimdjson.a | grep error_codes
---------------- S __ZN8simdjson8internal11error_codesE

and the failing tests have linked with libsimdjson.a

$ cat bazel-bin/tests/config_test-2.params
-o                                                                                                                                                                                                                                    
bazel-out/darwin_arm64-opt/bin/tests/config_test                                                                                                                                                                                      
-Xlinker                                                                                                                                                                                                                              
-rpath
-Xlinker
@loader_path/../_solib_darwin_arm64/
-Xlinker
-rpath
-Xlinker
@loader_path/config_test.runfiles/cris-core/_solib_darwin_arm64/
-Lbazel-out/darwin_arm64-opt/bin/_solib_darwin_arm64
bazel-out/darwin_arm64-opt/bin/tests/_objs/config_test/config_test.o
-llibconfig
bazel-out/darwin_arm64-opt/bin/external/simdjson/libsimdjson/lib/libsimdjson.a
-ltests_Slibcris_Ugtest_Umain
-llibsignal
-llibtimer
-llibutils
bazel-out/darwin_arm64-opt/bin/external/fmt/libfmt/lib/libfmt.a
-lexternal_Scom_Ugithub_Ugoogle_Uglog_Slibglog
-lexternal_Scom_Ugithub_Ugflags_Ugflags_Slibgflags
-lexternal_Sgtest_Slibgtest
-undefined
dynamic_lookup
-headerpad_max_install_names
-Wall
-Werror
-Wextra
-Wno-fuse-ld-path
-Wall
-Werror
-Wextra
-Wno-fuse-ld-path
-Wall
-Werror
-Wextra
-Wno-fuse-ld-path
-Wall
-Werror
-Wextra
-Wno-fuse-ld-path
-Wall
-Werror
-Wextra
-Wno-fuse-ld-path
-Wall
-Werror
-Wextra
-Wno-fuse-ld-path
-lpthread
-pthread
-L/opt/homebrew/opt/openssl/lib
-L/opt/homebrew/lib
-flto=thin
-lc++     
-lm    

But it only happens on Mac OS (with Apple Clang toolchain), so this is not a blocker.

[xkszltl]

Please provide the entire bazel-bin/tests/config_test-2.params

[chenhao94]

@xkszltl updated the description

[chenhao94]

I guess it may be because the loader on the Mac OS does not load the symbols from the static library for the dynamic libraries. Seems a bug to me.

[xkszltl]

I guess it may be because the loader on the Mac OS does not load the symbols from the static library for the dynamic libraries. Seems a bug to me.

Are you trying to say A.dylib depends on B.a but doesn't includes everything from B.a? That's expected, A.dylib is not responsible for unused parts in B.a, dynamic lib shares the same linking logic with executable.

[xkszltl]

Linker cmd LGTM, wondering if it has anything to do with LTO. Recommend to try without LTO first.

That S in symbol table also looks suspicious.

From man nm:

Each symbol name is preceded by its value (blanks if undefined). Unless the -m option is specified, this value is followed by one of the following characters, representing the symbol type: U (undefined), A (absolute), T (text section symbol), D (data section symbol), B (bss section symbol), C (common symbol), - (for debugger symbol table entries; see -a below), S (symbol in a section other than those above), or I (indirect symbol). If the symbol is local (non-external), the symbol's type is instead represented by the corresponding lowercase letter. A lower case u in a dynamic shared library indicates a undefined reference to a private external in another module in the same library.

[chenhao94]

FYI, disabling lto does not help.

[chenhao94]

chenhao@haos-mac-mini ~/d/c/bazel-bin> nm -C external/simdjson/libsimdjson/lib/libsimdjson.a | grep error_codes
---------------- S __ZN8simdjson8internal11error_codesE
chenhao@haos-mac-mini ~/d/c/bazel-bin> nm -C libconfig.dylib | grep error_codes
                 U simdjson::internal::error_codes
chenhao@haos-mac-mini ~/d/c/bazel-bin> nm -C tests/config_test | grep error_codes

On Mac, the symbol in the dylib file is demangled, while the symbol in .a is mangled. Also, in the executable binary output, there are no such a symbol.

While on Linux, the output is the following:

root@88e7bcfc9359:~/.cache/bazel/_bazel_root/2cd59afb93b178a87ec7d73dc50fa44d/execroot/cris-core/bazel-out/aarch64-opt/bin# nm -C external/simdjson/libsimdjson/lib/libsimdjson.a | grep error_codes
00000000 T simdjson::internal::error_codes
root@88e7bcfc9359:~/.cache/bazel/_bazel_root/2cd59afb93b178a87ec7d73dc50fa44d/execroot/cris-core/bazel-out/aarch64-opt/bin# nm -C libconfig.so | grep error_codes
                 U simdjson::internal::error_codes
root@88e7bcfc9359:~/.cache/bazel/_bazel_root/2cd59afb93b178a87ec7d73dc50fa44d/execroot/cris-core/bazel-out/aarch64-opt/bin# nm -C tests/config_test | grep error_codes
0000000000209248 R simdjson::internal::error_codes

[xkszltl]

What if you remove -C? Are they the same before demangling?