ASan broken on Mac OS X 10.8 after merge with UBsan run-time

[Quuxplusone]

Bugzilla Link	PR23539
Status	RESOLVED FIXED
Importance	P normal
Reported by	Hans Wennborg (hans@chromium.org)
Reported on	2015-05-15 13:08:40 -0700
Last modified on	2015-07-15 19:22:47 -0700
Version	unspecified
Hardware	PC Linux
CC	ganna@apple.com, glider@google.com, llvm-bugs@lists.llvm.org, mracek@apple.com, nicolasweber@gmx.de, t.p.northover@gmail.com, vonosmas@gmail.com
Fixed by commit(s)
Attachments
Blocks
Blocked by
See also

After the merge with UBSan run-time r233860, r233861, etc., the ASan run-time
no longer builds or runs on Mac OS X 10.8.

In Chromium, we have builders running ASan-ified tests on 10.8.5. They are now
failing with an unresolved symbol: (from [1])

dyld: Symbol not found: __ZTISt9type_info
  Referenced from: /Volumes/data/b/build/slave/mac_chromium_asan_rel_ng/build/src/gpu/../out/Release/libclang_rt.asan_osx_dynamic.dylib
  Expected in: /usr/lib/libc++.1.dylib
 in /Volumes/data/b/build/slave/mac_chromium_asan_rel_ng/build/src/gpu/../out/Release/libclang_rt.asan_osx_dynamic.dylib

It seems this symbol is expected to be available in either
/usr/lib/libc++.dylib or /usr/lib/libc++abi.dylib (this is apparently a known
tricky symbol, and the run-time is linked against both; see e.g.
http://reviews.llvm.org/D6960).

However, the symbol is exported from neither of these on our buildbots:

$ nm /usr/lib/libc++.dylib | grep __ZTISt9type_info
$ nm /usr/lib/libc++abi.dylib | grep __ZTISt9type_info
000000000002eb40 s __ZTISt9type_info

(Note the lower-case 's' means it's an internal symbol.)

Since this symbol isn't available, can we make ASan not rely on it?

$ sw_vers -productVersion
10.8.5
$ uname -a
Darwin vm723-m4.golo.chromium.org 12.6.0 Darwin Kernel Version 12.6.0: Wed Dec
17 19:11:40 PST 2014; root:xnu-2050.48.15~1/RELEASE_X86_64 x86_64

 [1] http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_asan_rel_ng/builds/223/steps/compile%20%28with%20patch%29/logs/stdio

[Quuxplusone]

Example of trying to build asan on 10.8.5:

$ git clone http://llvm.org/git/llvm.git
$ cd llvm/projects/
$ git clone http://llvm.org/git/compiler-rt.git
$ cd ..
$ mkdir build
$ cd build
$ CXXFLAGS="-stdlib=libc++" cmake -GNinja -DCMAKE_BUILD_TYPE=Release -
DLLVM_ENABLE_ASSERTIONS=ON ..
$ ninja lib/clang/3.7.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib

[73/73] Linking CXX shared library lib...win/libclang_rt.asan_osx_dynamic.dylib
FAILED: : && /usr/bin/c++  -stdlib=libc++  -fPIC -fvisibility-inlines-hidden -
Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-
initializers -pedantic -Wno-long-long -Wcovered-switch-default -std=c++11 -
fcolor-diagnostics -Wall -std=c++11 -O3  -arch x86_64 -arch i386 -dynamiclib -
Wl,-headerpad_max_install_names -mmacosx-version-min=10.7 -stdlib=libc++ -lc++ -
lc++abi -isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk
-o lib/clang/3.7.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib -install_name
@rpath/libclang_rt.asan_osx_dynamic.dylib projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_allocator.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_activation.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_debugging.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_fake_stack.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_flags.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_globals.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_interceptors.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_linux.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_mac.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_malloc_linux.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_malloc_mac.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_malloc_win.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_poisoning.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_posix.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_report.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_rtl.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_stack.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_stats.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_suppressions.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_thread.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_win.cc.o projects/compiler-
rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_new_delete.cc.o projects/compiler-
rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_linux.cc.o
projects/compiler-
rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_mac.cc.o
projects/compiler-
rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_win.cc.o
projects/compiler-
rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_type_test.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_allocator.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_common.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_deadlock_detector1.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_deadlock_detector2.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_flags.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_flag_parser.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_libc.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_libignore.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_linux.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_mac.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_persistent_allocator.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_platform_limits_linux.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_platform_limits_posix.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_posix.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_printf.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_common.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_freebsd.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_linux.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_mac.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stackdepot.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace_printer.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_suppressions.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_libbacktrace.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_mac.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_win.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_tls_get_addr.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_thread_registry.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_win.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_common_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_coverage_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_coverage_mapping_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_linux_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_posix_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stoptheworld_linux_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_posix_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_process_libcdep.cc.o
projects/compiler-
rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_unwind_posix_libcdep.cc.o
projects/compiler-rt/lib/lsan/CMakeFiles/RTLSanCommon.osx.dir/lsan_common.cc.o
projects/compiler-
rt/lib/lsan/CMakeFiles/RTLSanCommon.osx.dir/lsan_common_linux.cc.o
projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_diag.cc.o
projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_init.cc.o
projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_flags.cc.o
projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_handlers.cc.o
projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_value.cc.o
projects/compiler-
rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_handlers_cxx.cc.o
projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash.cc.o
-Wl,-rpath,@executable_path/../lib && :
Undefined symbols for architecture x86_64:
  "typeinfo for __cxxabiv1::__class_type_info", referenced from:
      __ubsan::checkDynamicType(void*, void*, unsigned long) in ubsan_type_hash.cc.o
      isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) in ubsan_type_hash.cc.o
      findBaseAtOffset(__cxxabiv1::__class_type_info const*, long) in ubsan_type_hash.cc.o
  "typeinfo for __cxxabiv1::__si_class_type_info", referenced from:
      isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) in ubsan_type_hash.cc.o
      findBaseAtOffset(__cxxabiv1::__class_type_info const*, long) in ubsan_type_hash.cc.o
  "typeinfo for __cxxabiv1::__vmi_class_type_info", referenced from:
      isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) in ubsan_type_hash.cc.o
      findBaseAtOffset(__cxxabiv1::__class_type_info const*, long) in ubsan_type_hash.cc.o
  "typeinfo for std::type_info", referenced from:
      __ubsan::checkDynamicType(void*, void*, unsigned long) in ubsan_type_hash.cc.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

[Quuxplusone]

Note that this is also pretty different from how you can normally build things on OS X: You can normally use (say) the OS X 10.10 SDK and as long as your deployment target is set to 10.7 (or whatever) your binary will run on 10.7. With this dependency, this no longer works. If you build with the 10.8 SDK, things won't build but if you build with the 10.10 SDK they will build and then not run.

This seems like a fairly serious problem.

[Quuxplusone]

(In reply to comment #2)
> Note that this is also pretty different from how you can normally build
> things on OS X:   You can normally use (say) the OS X 10.10 SDK and as long
> as your deployment target is set to 10.7 (or whatever) your binary will run
> on 10.7. With this dependency, this no longer works. If you build with the
> 10.8 SDK, things won't build but if you build with the 10.10 SDK they will
> build and then not run.

Why? Is it because __ZTISt9type_info is exported only from newer version of
libc++abi (it's in libcxx/lib/libc++abi2.exp, but not in
libcxx/lib/libc++abi.exp)?

> This seems like a fairly serious problem.

So, the problem is that we have old libc++/libc++abi either in SDK, or on the
machine, and they don't export proper symbols? That's tricky. I don't see how
we can work around this in an environment where we don't use fresh
libcxx/libcxxabi from trunk... except for disabling abi-relevant parts of ASan
and UBSan on Mac OS X for good.

Note tat this problem is not ASan specific, same will happen for UBSan.

[Quuxplusone]

(In reply to comment #3)
> Note tat this problem is not ASan specific, same will happen for UBSan.

Right, it's not ASan that's the problem, it's UBSan. But now UBSan run-time is
merged into ASan run-time, so it became a problem for us.

We've never used UBSan on Mac in Chromium, so this didn't come up until the
merge.

[Quuxplusone]

IIUC we can now fail to build ASan/UBSan runtime on some Mac OS X platforms, or build it correctly, but fail to run executables linked against it?

I'd say, we should just claim problematic -fsanitize=vptr which depends on libc++abi "unsupported" on Mac OS X, and exclude it from the build (which is a sad thing to do).

Kuba, Anna, what do you think?

[Quuxplusone]

We discussed this a bit today.

The ideal user experience in my opinion:

* ASan continues working like it does today. If you use a -mmacosx-version-min=
less than 10.7, the driver produces an error message

* UBSan requires 10.9+ and the driver errors if -mmacosx-version-min is less
than 10.9

Getting this right probably requires having separate dylibs for the asan and
the asan+ubsan runtime.

[Quuxplusone]

I am leaning to being more aggressive with loosing backward deployment. Do we know how many people are deploying ASan to 10.8 or earlier? Are we Ok with dropping 10.8 now?

(If we'll be Ok with dropping it in a few months, marking -fsanitize=vptr as "unsupported" on Mac OS X would be a fine temporary solution.)

[Quuxplusone]

Our ASan bots are on 10.7 and will stay there for at least a few more months. (We don't have any plans for upgrading them at the moment.)

[Quuxplusone]

Alexey, have you had a chance to look at this? Working around this is by far our largest and the only behavior-changing downstream patch when we build clang.

[Quuxplusone]

Sorry for delay, looking. It's only vptr sanitizer what's causing problems (C++-
specific parts of UBSan runtime), and it's not expected to work before 10.9.
I think we should do the following:

1) When we build compiler-rt we should actually *look* to -mmacosx-version-min
passed to CMAKE_CXX_FLAGS. If it's specified, we should use it. If it's
not specified, we take the default from MACOSX_DEPLOYMENT_TARGET. Now, when we
know the target, if it's less than 10.7 we refuse to build anything. If it's
10.7 or 10.8, we exclude certain parts of UBSan from build. If it's 10.9+, we
build everything.

2) When we're linking the program with ASan *or* UBSan, the driver looks at
deployment target. If it's less than 10.7, it reports an error. Otherwise we
just link with corresponding runtime. If the runtime was built with
-mmacosx-version-min=10.9, but we use it to link program with
-mmacosx-version-min=10.7, it would crash, but that's fine.

3) We silently exclude -fsanitize=vptr instrumentation from -fsanitize=undefined
on versions earlier than 10.9. If it's enabled explicitly, we report an error.

Let me know if you're OK with this plan.

[Quuxplusone]

An update on this: I have tried to land http://reviews.llvm.org/D10467 (not
Darwin-specific) that would let ToolChain object decide which sanitizers
it supports. Later we could use that to restrict certain sanitizers on
older versions of MacOS / iOS.

Turns out that it would require quite some yak shaving: Darwin ToolChain detects
the actual platform name and version we target in a terrible way and too late
(in Darwin::AddDeploymentTarget()), while now we would need this information
earlier - as soon as we construct the ToolChain.

I will now try to address FIXMEs in that code and see if we can learn the
platform from ArgList right in Darwin toolchain constructor.

[Quuxplusone]

>Our ASan bots are on 10.7 and will stay there for at least a few more months.
(We
>don't have any plans for upgrading them at the moment.)

OS X adoption rates are very high; for example, 55% are already on 10.10. I
suspect only a small percentage of users are running on < 10.9. Is there a
reason not to upgrade the buildbots 10.10 or 10.9 in the long term?

[Quuxplusone]

FYI I've mailed http://reviews.llvm.org/D10621, and would be grateful if you could verify it fixes the problems you observe in Chromium.

[Quuxplusone]

(In reply to comment #12)
> >Our ASan bots are on 10.7 and will stay there for at least a few more
months. (We
> >don't have any plans for upgrading them at the moment.)
>
> OS X adoption rates are very high; for example, 55% are already on 10.10. I
> suspect only a small percentage of users are running on < 10.9. Is there a
> reason not to upgrade the buildbots 10.10 or 10.9 in the long term?

This is not true in our experience. When a new OS X version is released, many
initial adoption is pretty fast but many people stay around on their current
version for a long time. (And for some reason 10.6 is very popular, it's often
the 2nd-most popular release after the current one.)

samsonov: I'm trying to look at this now. Doing a build of clang with
MACOSX_DEPLOYMENT_TARGET=10.6 now dies in compiler-rt's cmake with

  Too old OS X version: 10.6

It looks like http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-
20150622/284048.html added MACOSX_DEPLOYMENT_TARGET as the default deployment
version for compiler-rt. That's not correct. MACOSX_DEPLOYMENT_TARGET at clang
build time specifies which version of OS X the clang I'm building should run
on, not the version of OS X the binaries produced by that clang should run on.

I need to be able to build a clang that runs on 10.6. The asan'd binaries it
produces can assume 10.7, but the compiler itself needs to be able to run on
10.6, and it needs to be able to produce (non-asan) binaries for 10.6.

[Quuxplusone]

Adding a

  set(SANITIZER_MIN_OSX_VERSION "10.7")

in that file at least makes things build. So making that variable overridable
from the cmake invocation might be enough.

(local hack I'm using to test:

Index: CMakeLists.txt
===================================================================
--- CMakeLists.txt  (revision 241602)
+++ CMakeLists.txt  (working copy)
@@ -305,6 +305,7 @@
       list(APPEND SANITIZER_COMMON_SUPPORTED_OS iossim)
     endif()
   endif()
+  set(SANITIZER_MIN_OSX_VERSION "10.7")
   if(SANITIZER_MIN_OSX_VERSION VERSION_LESS "10.7")
     message(FATAL_ERROR "Too old OS X version: ${SANITIZER_MIN_OSX_VERSION}")
   endif()

)

[Quuxplusone]

With this change, things seem to work (I built a clang package and deployed it to our bots -- it's only been in for 40 minutes so far, but the bot that had issues with __ZTISt9type_info is happy).

So if you could add some way to override SANITIZER_MIN_OSX_VERSION to 10.7 via some cmake -D parameter, we'd be all set.

[Quuxplusone]

I've implemented your suggestion in r242363. I didn't aniticipate that use case (you use new Mac OS X to build Clang that would run on 10.6+, but would only be expected to produce working sanitizer binaries on 10.7+).

Let me know if there are more problems here.