Undefined references to nearly all TBB symbols when built with `-fvisibility=hidden`

[rob-p]

Hi,

First, thank you for your work on the excellent TBB library. We've been using it in some of the software in our lab for a number of years and it's a great resource. Specifically, we use TBB extensively in our tool salmon and we also use it in our tool pufferfish.

Recently, I have been trying to rid our code of the last deprecated remnants of the older TBB library, and to upgrade to the oneAPI TBB interfaces and idioms. While the code changes themselves were relatively painless, the challenges we have faced in building using the new library have been anything but.

We make use of holy build box to build widely distributable linux executables for our tools. This allows us to link against most of our libraries statically, and to have a very old libc requirement. This worked great up until the newer versions of oneAPI TBB. However, the build now breaks in the final linking step, finding undefined references to basically all symbols.

I tried everything I could think of, and then eventually discovered that these missing symbol issues occur when I'm not even trying to build our code, but rather, when I am even building the TBB examples in this environment itself.

To reproduce this issue, one can simply do the following:

• Create a (Docker) image for holy build box
• Enter an interactive environment in the above docker image (e.g. with docker run -it <image name> bash)
• Execute the following commands:
  • fetch the latest TBB curl -LJO https://github.com/oneapi-src/oneTBB/archive/refs/tags/v2021.5.0.tar.gz
  • tar xzvf v2021.5.0.tar.gz
  • cd oneTBB-2021.5.0
  • mkdir build && cd build
  • source /hbb_shlib/activate (activate the development environment of holy build box)
  • cmake ..
  • make

This will build the shared libraries without a problem, but when it attempts to build the examples, they will all fail with linker errors. Here is an example of what I see:

[ 16%] Linking C executable ../gnu_9.3_cxx17_64_relwithdebinfo/test_malloc_pure_c
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/test_malloc_pure_c.dir/tbbmalloc/test_malloc_pure_c.c.o: in
 function `MyExit':
/oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:50: undefined reference to `scalable_malloc'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:52: undefined ref
erence to `scalable_free'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/test_malloc_pure_c.dir/tbbmalloc/test_malloc_pure_c.c.o: in
 function `main':
/oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:64: undefined reference to `scalable_malloc'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:68: undefined ref
erence to `scalable_free'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:64: undefined ref
erence to `scalable_malloc'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:68: undefined ref
erence to `scalable_free'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:75: undefined ref
erence to `scalable_free'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:72: undefined ref
erence to `scalable_malloc'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:92: undefined ref
erence to `scalable_free'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:93: undefined ref
erence to `scalable_malloc'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /oneTBB-2021.5.0/test/tbbmalloc/test_malloc_pure_c.c:80: undefined ref
erence to `scalable_free'

and on and on like this. Strangely, these symbols seem to be present in the shared libraries (inspected with strings and c++filt), but they are simply not found when linking against these libraries. At this point, I've wasted a few days trying to get this to build in the holy build box image — updating that image several times to the most recent version — to no avail. I will mention that, with the old versions of TBB (e.g. the 2019 or 2020 versions), the build works. Only starting with the new oneAPI TBB versions from 2021.1 forward does linking seem to fail.

I would appreciate any insight you might have into this issue, that is really blocking us from adopting an updated and non-deprecated version of TBB.

Thanks, Rob

[FooBarWidget]

This is because Holy Build Box sets CFLAGS/CXXFLAGS to -g -O2 -fvisibility=hidden -I/hbb_shlib/include by default. Proper usage of -fvisibility=hidden requires help from the library in question: it has to explicitly mark exportable symbols as such (with __attribute__((visibility ("default"))). It appears OneTBB doesn't do that.

Removing -fvisbility=hidden from CFLAGS/CXXFLAGS solves the symbol linking problem. But of course, TBB still exports too many symbols.

[rob-p]

Thanks for the quick response, @FooBarWidget! What is the recommended way to remove this flag in the HBB environment used for building?

[FooBarWidget]

Override CFLAGS/CXXFLAGS. They're just environment variables.

[alexey-katranov]

oneTBB relies on .def files to export only required systems. It seems they are not used under your build environment. To avoid all symbols exporting, either .def files should be used or include/oneapi/tbb/detail/_export.h should be extended to specify the required visibility for symbols.

[anton-potapov]

@FooBarWidget , @rob-p I was able to reproduce the link errors with -fvisibility=hidden flag applied.

@alexey-katranov it seems that .def files are not enough, as they only pass in already exported symbols, but does not make them exported

[FooBarWidget]

Export.h should define _EXPORT as __attribute__((visibility("default"))) on Unix, instead of as nothing.

[alexey-katranov]

Yes, #786 tries to addresses the issue.

[mika-fischer]

Please consider making exporting of symbols configurable or disable exporting when comping tbb as a static library.

[arunparkugan]

@anton-potapov is this issue still relevant?

[isaevil]

@arunparkugan it is, there is a linked PR to resolve this issue.