search

google / sanitizers

AddressSanitizer, ThreadSanitizer, MemorySanitizer
Other
11.46k stars 1.03k forks source link

Asan does not perform as expected when intrumenting a lib that uses Gnulib #1079

Open noloader opened 5 years ago

noloader commented 5 years ago

I've been trying to test GnuTLS and several other GNU packages under Asan. I dutifully build all dependent libraries with Asan, too. Everything gets built with LDFLAGS of -L<path> -Wl,-R,<path> -Wl,--enable-new-dtags, so everything will find the proper libraries and allow RUNPATH overrides via LD_LIBRARY_PATH.

One of the dependent libraries is GNU's IDN, which handles internationalized domain names. IDN passes its tests in non-instrumented builds. IDN is failing its tests under Asan.

IDN has three test runners. The first two test runners test libidn.so functionality, and they test OK. The third test runner tests Gnulib, which IDN uses. The Gnulib testing fails with the error shown below. I can duplicate the failure on both Debian 9 and Fedora 29.

$ cat libidn-1.35/tests/test-suite.log
===========================================
   GNU Libidn 1.35: tests/test-suite.log
===========================================

# TOTAL: 16
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  16
# XPASS: 0
# ERROR: 0

FAIL: tst_stringprep
====================

==21449==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
FAIL tst_stringprep (exit status: 1)

FAIL: tst_punycode
==================

==21455==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
FAIL tst_punycode (exit status: 1)

<Remaining 14 tests have same message>

I spent the better part of two days attempting to straighten out LD_LIBRARY_PATHs and potential LD_PRELOAD problems (I could not find use of LD_PRELOAD in this instance).

FIRST PROBLEM: It appears the message "ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD" is incorrect.

The bs message really pisses me off because it took me down a rabbit hole and wasted two days of my time, wasted the time of folks on the IDN mailing list, and wasted the time of folks on the Gnulib mailing list.

Please provide an accurate message during failures.

SECOND PROBLEM: It took me a while to get to this point because I was chasing Unicorns... It appears Asan is missing some symbols needed to properly intercept Gnulib.

$ LD_DEBUG=libs tests/tst_stringprep
     21696:     find library=libasan.so.3 [0]; searching
     21696:      search path=/var/sanitize/lib/tls/x86_64:/var/sanitize/lib/tls:/var/sanitize/lib/x86_64:/var/sanitize/lib:/home/build/libidn-1.35/lib/.libs/tls/x86_64:/home/build/libidn-1.35/lib/.libs/tls:/home/build/libidn-1.35/lib/.libs/x86_64:/home/build/libidn-1.35/lib/.libs    (RUNPATH from file tests/tst_stringprep)
     21696:       trying file=/var/sanitize/lib/tls/x86_64/libasan.so.3
     21696:       trying file=/var/sanitize/lib/tls/libasan.so.3
     21696:       trying file=/var/sanitize/lib/x86_64/libasan.so.3
     21696:       trying file=/var/sanitize/lib/libasan.so.3
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/tls/x86_64/libasan.so.3
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/tls/libasan.so.3
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/x86_64/libasan.so.3
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/libasan.so.3
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/usr/lib/x86_64-linux-gnu/libasan.so.3
     21696:
     21696:     find library=libidn.so.12 [0]; searching
     21696:      search path=/home/build/libidn-1.35/lib/.libs (RUNPATH from file tests/tst_stringprep)
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/libidn.so.12
     21696:
     21696:     find library=libdl.so.2 [0]; searching
     21696:      search path=/home/build/libidn-1.35/lib/.libs (RUNPATH from file tests/tst_stringprep)
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/libdl.so.2
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/lib/x86_64-linux-gnu/libdl.so.2
     21696:
     21696:     find library=libpthread.so.0 [0]; searching
     21696:      search path=/home/build/libidn-1.35/lib/.libs (RUNPATH from file tests/tst_stringprep)
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/libpthread.so.0
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/lib/x86_64-linux-gnu/libpthread.so.0
     21696:
     21696:     find library=libc.so.6 [0]; searching
     21696:      search path=/home/build/libidn-1.35/lib/.libs (RUNPATH from file tests/tst_stringprep)
     21696:       trying file=/home/build/libidn-1.35/lib/.libs/libc.so.6
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/lib/x86_64-linux-gnu/libc.so.6
     21696:
     21696:     find library=librt.so.1 [0]; searching
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/lib/x86_64-linux-gnu/librt.so.1
     21696:
     21696:     find library=libstdc++.so.6 [0]; searching
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6
     21696:
     21696:     find library=libm.so.6 [0]; searching
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/lib/x86_64-linux-gnu/libm.so.6
     21696:
     21696:     find library=libgcc_s.so.1 [0]; searching
     21696:      search cache=/etc/ld.so.cache
     21696:       trying file=/lib/x86_64-linux-gnu/libgcc_s.so.1
     21696:
     21696:     calling init: /lib/x86_64-linux-gnu/libpthread.so.0
     21696:
     21696:     calling preinit: tests/tst_stringprep
     21696:
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_printf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_sprintf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_snprintf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_fprintf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_vprintf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_vsprintf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_vsnprintf (fatal)
     21696:     /usr/lib/x86_64-linux-gnu/libasan.so.3: error: symbol lookup error: undefined symbol: __isoc99_vfprintf (fatal)
     21696:
     ...

libasan.so.3 is from Debian 9.9. On Fedora 28 and 29 the culprit is libasan.so.5.

I see a similar report at Issue 957, Initialization failure with long object paths, but I am not clear how to work around it. Also, the 957 issue seems to indicate 32-bit platforms, but I can reproduce on 64-bit platforms.

kcc commented 5 years ago

I guess this is happening with gcc, where the default for asan is to use shared library. With clang, the default is to use static library and this problem should not exist. Please try clang, or ask someone in the gcc land to help.

yugr commented 5 years ago

Please provide an accurate message during failures.

What is inaccurate about the message exactly? Please try to provide an MVCE.

FruitClover commented 5 years ago

Based on your log, standalone run LD_DEBUG=libs tests/tst_stringprep is ok (interceptions are initialized after ...ASan runtime does not come first in initial library list ... check, runtime library list order is correct here), so you have a problem with testsuite runner environment.

chefmax commented 5 years ago

@noloader so, LD_PRELOAD'ing ASan shared library didn't help you?

And regarding second problem (with missed symbols): actually, these __isoc99_vfprintf symbols were always undefined and honestly I don't know why ASan tries to intercept them. This error message is confusing, but these missing symbols are not a problem at all.