compiler-rt is built with the wrong toolchain (bootstrap compiler rather than built clang), linking fails when using clang older than 3.5 (eg: Mavericks and earlier Xcodes)

[Quuxplusone]

Bugzilla Link	PR25753
Status	REOPENED
Importance	P normal
Reported by	Jeremy Huddleston Sequoia (jeremyhu@apple.com)
Reported on	2015-12-05 12:26:50 -0800
Last modified on	2018-06-11 20:16:29 -0700
Version	unspecified
Hardware	Macintosh MacOS X
CC	chris.bieneman@me.com, dvyukov@google.com, llvm-bugs@lists.llvm.org, mracek@apple.com, nicolasweber@gmx.de, vitalybuka@google.com
Fixed by commit(s)
Attachments
Blocks
Blocked by
See also	PR25885

Updating from r253905 to r254705 fails on Mountain Lion and Mavericks (and
possibly others as well):

https://build.macports.org/builders/buildports-mtln-x86_64/builds/26284/steps/compile/logs/stdio/text

https://build.macports.org/builders/buildports-mavericks-x86_64/builds/15087/steps/compile/logs/stdio/text

On Mountain Lion, we were using OSS clang 3.4.  On Mavericks, we were using
Xcode 6.2 (clang 600.0.57).  The failure is in linking
libclang_rt.tsan_osx_dynamic.dylib

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
-pipe -Os -arch x86_64 -stdlib=libc++  -fPIC -fvisibility-inlines-hidden -Wall -
W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-
initializers -pedantic -Wno-long-long -Wcovered-switch-default -std=c++11 -Wall
-std=c++11 -g -arch x86_64 -arch x86_64h -dynamiclib -Wl,-
headerpad_max_install_names  -stdlib=libc++ -lc++ -lc++abi -mmacosx-version-
min=10.9 -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk
-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,@loader_path -arch
x86_64 -o
../../../../lib/clang/3.8.0/lib/darwin/libclang_rt.tsan_osx_dynamic.dylib -
install_name @rpath/libclang_rt.tsan_osx_dynamic.dylib
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_clock.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_flags.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_fd.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_ignoreset.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_interceptors.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_interface_ann.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_interface_atomic.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_interface.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_interface_java.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_malloc_mac.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_md5.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_mman.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_mutex.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_mutexset.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_report.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_rtl.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_rtl_mutex.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_rtl_report.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_rtl_thread.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_stack_trace.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_stat.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_suppressions.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_symbolize.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_sync.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_interceptors_mac.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_libdispatch_mac.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_platform_mac.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_platform_posix.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_new_delete.cc.o
CMakeFiles/clang_rt.tsan_osx_dynamic.dir/rtl/tsan_rtl_amd64.S.o
../interception/CMakeFiles/RTInterception.osx.dir/interception_linux.cc.o
../interception/CMakeFiles/RTInterception.osx.dir/interception_mac.cc.o
../interception/CMakeFiles/RTInterception.osx.dir/interception_win.cc.o
../interception/CMakeFiles/RTInterception.osx.dir/interception_type_test.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_allocator.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_common.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_deadlock_detector1.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_deadlock_detector2.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_flags.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_flag_parser.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_libc.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_libignore.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_linux.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_mac.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_persistent_allocator.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_platform_limits_linux.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_platform_limits_posix.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_posix.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_printf.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_common.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_freebsd.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_linux.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_mac.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stackdepot.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace_printer.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_suppressions.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_libbacktrace.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_mac.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_win.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_tls_get_addr.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_thread_registry.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_win.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_common_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_coverage_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_coverage_mapping_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_linux_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_posix_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_stacktrace_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_stoptheworld_linux_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_symbolizer_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_symbolizer_posix_libcdep.cc.o
../sanitizer_common/CMakeFiles/RTSanitizerCommonLibc.osx.dir/sanitizer_unwind_linux_libcdep.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_diag.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_init.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_flags.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_handlers.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_value.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_handlers_cxx.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash_itanium.cc.o
../ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash_win.cc.o -Wl,-
rpath,@executable_path/../lib
Undefined symbols for architecture x86_64:
  "___sync_fetch_and_add_16", referenced from:
      ___tsan_atomic128_fetch_add in tsan_interface_atomic.cc.o
  "___sync_fetch_and_and_16", referenced from:
      ___tsan_atomic128_fetch_and in tsan_interface_atomic.cc.o
  "___sync_fetch_and_or_16", referenced from:
      ___tsan_atomic128_fetch_or in tsan_interface_atomic.cc.o
  "___sync_fetch_and_sub_16", referenced from:
      ___tsan_atomic128_fetch_sub in tsan_interface_atomic.cc.o
  "___sync_fetch_and_xor_16", referenced from:
      ___tsan_atomic128_fetch_xor in tsan_interface_atomic.cc.o
  "___sync_lock_test_and_set_16", referenced from:
      ___tsan_atomic128_exchange in tsan_interface_atomic.cc.o
...

[Quuxplusone]

I believe that in older builds, tsan wasn't being built by default, so this may not be a new issue, just one that was uncovered by starting to build tsan by default.

[Quuxplusone]

The autoconf-based build system used the built clang to compile compiler-rt. Is this issue happening because cmake is instead using the bootstrap compiler?

[Quuxplusone]

Yeah, that looks to be the case.

[Quuxplusone]

I'm seeing this on my bots too, even after a clobber build. How did you fix this?

[Quuxplusone]

I guess this is from r254603 (http://reviews.llvm.org/D15109) if your Xcode isn't new enough? (Where "new enough" means "really new")

Since this causes issues, can we either make cmake check if the installed Xcode is new enough and only build tsan on OS X if it is, or at least provide a toggle to disable tsan on OS X?

[Quuxplusone]

Is there a way to have cmake use the newly built clang to compile the runtime libraries, like we do with the autoconf build system?

[Quuxplusone]

Hm, maybe we can just upgrade from Xcode 6.2 to Xcode 7 on that bot of mine.

But in general it'd be good if we were less reckless about requiring latest Xcodes.

[Quuxplusone]

You can use just-built clang to build compiler-rt in CMake if you set LLVM_BUILD_EXTERNAL_COMPILER_RT=On.

[Quuxplusone]

There are two solutions for this. Either pass LLVM_BUILD_EXTERNAL_COMPILER_RT=On to CMake, or on trunk you can put compiler-rt in the LLVM runtimes subdirectory.

[Quuxplusone]

Sorry Chris, but that doesn't actually work.

I tested that with our llvm-3.8 port on Mountain Lion in MacPorts without success. The LLVM_BUILD_EXTERNAL_COMPILER_RT=ON option causes none of compiler_rt to be built at all.

compiler-rt has always been in llvm/projects/compiler-rt. I don't see a "runtimes" directory, so I assume that's what you meant. AFAIK, it has to be placed there and there isn't really another option. Not doing that will certainly cause failure. Perhaps I'm missing something there ?

The issue is that compiler_rt should be built, but it (and likely the rest of the runtime libs for that matter) using the just-built clang and not the compiler that was used to build llvm and clang. The old autoconf-based build system used the built clang, but the cmake build system does not.

[Quuxplusone]

The "runtimes" directory is new in LLVM 4.0, so you can see and test it on trunk (which is why my earlier comment said you can do that on trunk).

I would expect LLVM_BUILD_EXTERNAL_COMPILER_RT=On to work on 3.8 and 3.9, but likely not earlier. With that compiler-rt should be built correctly. Does the generated build file contain a "compiler-rt" target?

[Quuxplusone]

Obsolete?