Compilation issues with Clang on Linux

[jdumas]

Consider this minimal cmake project (based on the CMakeLists.txt from Instant Meshes):

cmake_minimum_required(VERSION 3.3)
project(foo)

add_executable(${PROJECT_NAME} main.cpp)

# Build TBB
set(TBB_BUILD_STATIC ON CACHE BOOL " " FORCE)
set(TBB_BUILD_SHARED OFF CACHE BOOL " " FORCE)
set(TBB_BUILD_TBBMALLOC OFF CACHE BOOL " " FORCE)
set(TBB_BUILD_TBBMALLOC_PROXY OFF CACHE BOOL " " FORCE)
set(TBB_BUILD_TESTS OFF CACHE BOOL " " FORCE)

add_subdirectory(tbb)
set_property(TARGET tbb_static tbb_def_files PROPERTY FOLDER "dependencies")
target_include_directories(tbb_static PUBLIC tbb/include)

target_link_libraries(${PROJECT_NAME} PUBLIC tbb_static)
set_target_properties(${PROJECT_NAME} PROPERTIES CXX_STANDARD 11)

With the following main.cpp:

#include "tbb/blocked_range.h"
#include "tbb/parallel_for.h"
#include "tbb/task_scheduler_init.h"
#include <iostream>

int main(void) {
    auto func = [&](const tbb::blocked_range<uint32_t> &range) {
        for (uint32_t i = range.begin(); i < range.end(); ++i) {
            std::cout << i << std::endl;
        }
    };

    tbb::blocked_range<uint32_t> range(0u, 1024u, 10);
    tbb::parallel_for(range, func);

    return 0;
}

When I try to compile it with Clang on Linux:

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release ..

Then compilation fails with the following error:

In file included from /home/jdumas/downloads/test_tbb/tbb/src/tbb/arena.cpp:21:
In file included from /home/jdumas/downloads/test_tbb/tbb/include/tbb/global_control.h:28:
/home/jdumas/downloads/test_tbb/tbb/include/tbb/tbb_stddef.h:119:10: fatal error: 'cstddef' file not found
#include <cstddef>      /* Need size_t and ptrdiff_t */
         ^~~~~~~~~

I figure there is a missing path to Clang's libc++ headers that ought to be dragged along the way with the target tbb_static, but I'm not sure exactly how this should be set up.

[wjakob]

This issue is not related to this project -- your Clang install is likely missing libc++.

[jdumas]

Ok I feel stupid now ... I thought libc++ was installed because I already had a bunch of stuff under /usr/lib/clang/4.0.1/include/, but apparently those are separate entitities ... Closing.

[jdumas]

Hmm, I think I spoke too soon. Now that I have libc++ installed, the compilation fails at link time with the following errors:

tbb/libtbb_static.a(scheduler.cpp.o): In function `tbb::internal::custom_scheduler<tbb::internal::DefaultSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*)':
/home/jdumas/test/tbb/src/tbb/scheduler.cpp:(.text._ZN3tbb8internal16custom_schedulerINS0_22DefaultSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_[_ZN3tbb8internal16custom_schedulerINS0_22DefaultSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_]+0x935): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/scheduler.cpp:(.text._ZN3tbb8internal16custom_schedulerINS0_22DefaultSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_[_ZN3tbb8internal16custom_schedulerINS0_22DefaultSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_]+0x93d): undefined reference to `std::rethrow_exception(std::exception_ptr)'
/home/jdumas/test/tbb/src/tbb/scheduler.cpp:(.text._ZN3tbb8internal16custom_schedulerINS0_22DefaultSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_[_ZN3tbb8internal16custom_schedulerINS0_22DefaultSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_]+0x9c5): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(scheduler.cpp.o): In function `tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*)':
/home/jdumas/test/tbb/src/tbb/scheduler.cpp:(.text._ZN3tbb8internal16custom_schedulerINS0_20IntelSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_[_ZN3tbb8internal16custom_schedulerINS0_20IntelSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_]+0x872): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/scheduler.cpp:(.text._ZN3tbb8internal16custom_schedulerINS0_20IntelSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_[_ZN3tbb8internal16custom_schedulerINS0_20IntelSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_]+0x87a): undefined reference to `std::rethrow_exception(std::exception_ptr)'
/home/jdumas/test/tbb/src/tbb/scheduler.cpp:(.text._ZN3tbb8internal16custom_schedulerINS0_20IntelSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_[_ZN3tbb8internal16custom_schedulerINS0_20IntelSchedulerTraitsEE18local_wait_for_allERNS_4taskEPS4_]+0x902): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(task_group_context.cpp.o): In function `tbb::internal::tbb_exception_ptr::allocate()':
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x284): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x28c): undefined reference to `std::exception_ptr::~exception_ptr()'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x2b5): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(task_group_context.cpp.o): In function `tbb::internal::tbb_exception_ptr::allocate(tbb::tbb_exception const&)':
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x304): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x30c): undefined reference to `std::exception_ptr::~exception_ptr()'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x335): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(task_group_context.cpp.o): In function `tbb::internal::tbb_exception_ptr::destroy()':
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x415): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(task_group_context.cpp.o): In function `tbb::task_group_context::~task_group_context()':
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0x70b): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(task_group_context.cpp.o): In function `tbb::task_group_context::reset()':
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0xa87): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(task_group_context.cpp.o): In function `tbb::task_group_context::register_pending_exception()':
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0xda4): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0xdae): undefined reference to `std::exception_ptr::~exception_ptr()'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0xdcc): undefined reference to `std::exception_ptr::~exception_ptr()'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0xddb): undefined reference to `std::exception_ptr::~exception_ptr()'
/home/jdumas/test/tbb/src/tbb/task_group_context.cpp:(.text+0xdec): undefined reference to `std::exception_ptr::~exception_ptr()'
tbb/libtbb_static.a(tbb_misc.cpp.o): In function `tbb::internal::throw_exception_v4(tbb::internal::exception_id)':
/home/jdumas/test/tbb/src/tbb/tbb_misc.cpp:(.text+0x2e4): undefined reference to `std::bad_alloc::bad_alloc()'
tbb/libtbb_static.a(tbb_misc.cpp.o): In function `tbb::bad_last_alloc::bad_last_alloc()':
/home/jdumas/test/tbb/src/tbb/tbb_misc.cpp:(.text._ZN3tbb14bad_last_allocC2Ev[_ZN3tbb14bad_last_allocC2Ev]+0x5): undefined reference to `std::bad_alloc::bad_alloc()'
tbb/libtbb_static.a(arena.cpp.o): In function `tbb::interface7::internal::task_arena_base::internal_execute(tbb::interface7::internal::delegate_base&) const':
/home/jdumas/test/tbb/src/tbb/arena.cpp:(.text+0x1b17): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/arena.cpp:(.text+0x1b1f): undefined reference to `std::rethrow_exception(std::exception_ptr)'
/home/jdumas/test/tbb/src/tbb/arena.cpp:(.text+0x1b34): undefined reference to `std::exception_ptr::~exception_ptr()'
/home/jdumas/test/tbb/src/tbb/arena.cpp:(.text+0x1cbc): undefined reference to `std::exception_ptr::exception_ptr(std::exception_ptr const&)'
/home/jdumas/test/tbb/src/tbb/arena.cpp:(.text+0x1cc4): undefined reference to `std::rethrow_exception(std::exception_ptr)'
/home/jdumas/test/tbb/src/tbb/arena.cpp:(.text+0x1cd9): undefined reference to `std::exception_ptr::~exception_ptr()'
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/foo.dir/build.make:96: foo] Error 1
make[1]: *** [CMakeFiles/Makefile2:68: CMakeFiles/foo.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

make VERBOSE=1 shows the following line being used:

/usr/lib/ccache/bin/clang++  -O3 -DNDEBUG  -rdynamic CMakeFiles/foo.dir/main.cpp.o  -o foo tbb/libtbb_static.a -lpthread -ldl

If I manually add the option -stdlib=libc++, the error is reduced to the following:

CMakeFiles/foo.dir/main.cpp.o: In function `tbb::interface9::internal::start_for<tbb::blocked_range<unsigned int>, main::$_0, tbb::auto_partitioner const>::run_body(tbb::blocked_range<unsigned int>&)':
/home/jdumas/test/main.cpp:(.text+0x607): undefined reference to `std::cout'
/home/jdumas/test/main.cpp:(.text+0x616): undefined reference to `std::ostream& std::ostream::_M_insert<unsigned long>(unsigned long)'
/home/jdumas/test/main.cpp:(.text+0x644): undefined reference to `std::ctype<char>::_M_widen_init() const'
/home/jdumas/test/main.cpp:(.text+0x65d): undefined reference to `std::ostream::put(char)'
/home/jdumas/test/main.cpp:(.text+0x665): undefined reference to `std::ostream::flush()'
/home/jdumas/test/main.cpp:(.text+0x67a): undefined reference to `std::__throw_bad_cast()'
CMakeFiles/foo.dir/main.cpp.o: In function `_GLOBAL__sub_I_main.cpp':
/home/jdumas/test/main.cpp:(.text.startup+0xc): undefined reference to `std::ios_base::Init::Init()'
/home/jdumas/test/main.cpp:(.text.startup+0x13): undefined reference to `std::ios_base::Init::~Init()'
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)

I have of course tried to replace /usr/lib/ccache/bin/clang++ by /bin/clang++, but it doesn't change anything in the error.

Any idea what flags could be missing? I tried several options to no avail. It compiles fine with g++.

[wjakob]

Looks like there is still some sort of issue with your compiler. Not finding std::cout etc. makes this fairly clear. I committed a patch just now (4c6c70969e03649af1acdeb90eada3efa5568343) which decouples using Clang and libc++ from another. So you should be able to link with Clang & classic libstdc++, let me know if that works.

[jdumas]

I'm running on archlinux with clang from the official repositories, and libc++ from the AUR. What is strange is that I can compile a hello world with libc++ from the command line:

#include <cstddef>
#include <iostream>

int main(void) {
    std::cout << "hello" << std::endl;
    return 0;
}

Compiled with the following command:

clang++ main.cpp -stdlib=libc++ -lc++abi

Works perfectly fine. I will try to compile larger projects with libc++ to see if the problem is specific to tbb or not.

[wjakob]

I'm assuming that this is some kind of setup issue on your machine -- we routinely build with this repo using Clang on Linux, and no such issues appear. If you want me to take a look, I'll need something reproducible.

[jdumas]

There's nothing special about my setup, I'm simply running archlinux. libc++ isn't available in a precompiled form on the official repository, so we have to build it from the AUR. I haven't fiddled with clang+tbb in a while but I'll let you know if I do.

[wjakob]

Clang & LLVM are usually built jointly along with libc++, llldb, etc.

Potentially the issue is installing an external libc++ that wasn't compiled as part of the same build process.

[jdumas]

Well on archlinux for some reason libc++ is not part of the deal :D (clang, llvm, lldb are all available on the official repo, but by default it will use the libstd++ implementation).

[wjakob]

Aha, I see - one potential workaround would be to detect if libc++ is truly missing, and to have Clang fall back to libstdc++ in that case. (Though in my experience libstdc++ is pretty terrible when used with Clang)

[jdumas]

Pretty terrible as in, performances suck? Or leads to bugs, etc.?

[wjakob]

As in: programs don't compile or don't link properly.