Address Sanitizer fails to intercept function in shared library opened with RTLD_DEEPBIND

[Keno]

I just found out that if an application opens a shared library (both built with address sanitizer) using RTLD_DEEPBIND, address sanitizer will fail to intercept the functions called from the so opened shared library. In my particular case, this manifested itself as a strdup which, though originally skipping any interceptors and going into libc, would eventually call asan's allocator. However the same did not happen for free (since it went straight into glibc's free), so it failed to intercept free (or rather there was an allocator mismatch). I am not familiar enough with glibc to say whether this can be fixed or not, but perhaps we could drop the RTLD_DEEPBIND flag in our dlopen interceptor and issue a warning to the user.

[eugenis]

Interesting. I think this has no chance of working with static-libasan, so we should either remove DEEPBIND or complain about it or both. It works with -shared-libasan with clang but not with gcc, because gcc does not add DT_NEEDED for libasan to shared libraries.

[kcc]

On Mon, Oct 12, 2015 at 9:30 AM, Evgeniy Stepanov notifications@github.com wrote:

Interesting. I think this has no chance of working with static-libasan, so we should either remove DEEPBIND or complain

I'd say we need to complain and exit. (We may of course, have a flag to guard the logic here, but that's on overkill)

about it or both. It works with -shared-libasan with clang but not with gcc, because gcc does not add DT_NEEDED for libasan to shared libraries.

— Reply to this email directly or view it on GitHub https://github.com/google/sanitizers/issues/611#issuecomment-147453237.

[yugr]

Does this boil down to detecting RTLD_DEEPBIND in dlopen interceptor?

[kcc]

I'd guess so.

[FerCampos]

Hi! I have an environment in which I am using an external SDK. This SDK consist of a shared library I link with, and which makes use of dlopen to open other libX.so with RTLD_DEEPBIND flag. I'm not interested at all on sanitize the SDK, I only want to sanitize my own code.

How should I proceed? I'm confused about the different -shared-libasan/-static-libasan options. As far as I understand, compiling with -static-libasan (GCC6) should inject all the sanitizer code in my binary and should have no effects on non-sanitized SDK, but I get an error on SDK trying to dlopen the .so: Couldn't load libXXX.so: System error code(-2): libXXX.so: cannot open shared object file: No such file or directory

Using LD_LIBRARY_PATH to allow the SDK finding its libXXX.so I get the famous _int_free SEGV ASAN error. Seems like the SDK is redefining any alloc/free function, so there is a mismatch on symbol resolution (SDK vs Asan).

Looks like linking with -fsanitize=address changes the order of symbols lookup/LD_PATH. I would need to isolate the SDK so it remains using libc or its own implementation (I don't know how internally works and I don't have access to source code neither).

I want this behavior https://github.com/google/sanitizers/issues/871#issuecomment-337896857, but I'm not able to make the SDK run when my binary is using asan.

I am open to use blacklist, suppression list, attribute((no_sanitize("address"))) or any other option, but I would like to understand the problem.

Any hints? Thanks

[kcc]

Frankly, your best choice is to contact the vendor for the SDK and request that they support asan in whatever environment this SDK is being used. We've seen cases in the past when a closed-source SDK did something extremely asan-hostile and there is very little we could do there.

If you can provide a small reproducer, where you replace the SDK with a tiny mock library, please file a separate bug and we'll take a look (but no promises, such problems are often hard to solve)

[jm4R]

On Mon, Oct 12, 2015 at 9:30 AM, Evgeniy Stepanov notifications@github.com wrote: I'd say we need to complain and exit. (We may of course, have a flag to guard the logic here, but that's on overkill)

Isn't it a good idea to add such flag? In code with a lot of modules this could help a lot as one wouldn't have to recompile some of them.

[faridosc]

I am getting this error:

==7741==You are trying to dlopen a /usr/share/icedtea-web/lib/haswell/IcedTeaPlugin.so shared library with RTLD_DEEPBIND flag which is incompatibe with sanitizer runtime (see https://github.com/google/sanitizers/issues/611 for details). If you want to run /usr/share/icedtea-web/lib/haswell/IcedTeaPlugin.so library under sanitizers please remove RTLD_DEEPBIND from dlopen flags.

[vrqq]

Interesting. I think this has no chance of working with static-libasan, so we should either remove DEEPBIND or complain about it or both. It works with -shared-libasan with clang but not with gcc, because gcc does not add DT_NEEDED for libasan to shared libraries.

Which version of clang could make -shared-libasan and DEEPBIND together? I have tried clang 11 and 12 with -shared-libsan or -shared-libasan but still got error in runtime: You are trying to dlopen a /usr/lib64/libssl.so shared library with RTLD_DEEPBIND flag which is incompatible with sanitizer runtime

Can I use [[no_sanitize("address")]] to solve it?

Thanks

===== btw: The problem I face, is some close source third library who depend on the other library which is not the same version on my host, so, denied the DEEPBIND is a problem for me. (for example loading both OpenSSL 1.0 and 1.1) I have to use dlopen(RTLD_LOCAL|RTLD_DEEPBIND) in my code, and dlmopen is also have a problem that some code not running on that namespace.

Maybe using blacklist is better than denied RTLD_DEEPBIND? Just warn and let user to add the relevant files into blacklist.

The other solution is to allow the RTLD_DEEPBIND to .so dynamic library which build with ASAN, so we can implement some shims easily. (such as https://github.com/yugr/Implib.so, https://github.com/dezgeg/libcapsule and other libGL shims)

[WilliamTambellini]

Note: asan from gcc6 was ok with DEEPBIND, it started to fail from gcc8 (or 7?). Has anyone retried with gcc11 ?

[janwilmans]

tried, gcc-11.2.0 same problem, if a third-party library uses RTLD_DEEPBIND to load a shared library building with TSAN will now make the application report an error and refuse to start .

[janwilmans]

would it be possible to accept the fact that the code in the shared library can't be checked? In my case any application that links with this closed sources library can't use sanitizers anymore... It would be nice if I could still check my own code?

[WilliamTambellini]

Same behavior with gcc11.3. +1 for @janwilmans proposal.

[AllanZyne]

I encountered an interesting false-positive. When dlopen(RTLD_DEEPBIND) a shared library which doesn't exist, Address Sanitizer will also die.

[Enna1]

I encountered an interesting false-positive. When dlopen(RTLD_DEEPBIND) a shared library which doesn't exist, Address Sanitizer will also die.

ASan add this check in dlopen interceptor, if flags contain RTLD_DEEPBIND then ASan will report and die.

if (flag & RTLD_DEEPBIND) {
    Report(
        "You are trying to dlopen a %s shared library with RTLD_DEEPBIND flag"
        " which is incompatible with sanitizer runtime "
        "(see https://github.com/google/sanitizers/issues/611 for details"
        "). If you want to run %s library under sanitizers please remove "
        "RTLD_DEEPBIND from dlopen flags.\n",
        filename, filename);
    Die();
  }

[janwilmans]

https://github.com/llvm/llvm-project/commit/76630d43f6081c657ac5be2406061d43334045dd added this check and it was well intentioned, however, if the library doesn't exist, there is no problem and the sanitizer still dies. Also the loaded library might never be actually be used (or its use is intentionally avoided, like we do) to allow you to still check your own code.

[AllanZyne]

I encountered an interesting false-positive. When dlopen(RTLD_DEEPBIND) a shared library which doesn't exist, Address Sanitizer will also die.

ASan add this check in dlopen interceptor, if flags contain RTLD_DEEPBIND then ASan will report and die.

if (flag & RTLD_DEEPBIND) {
    Report(
        "You are trying to dlopen a %s shared library with RTLD_DEEPBIND flag"
        " which is incompatible with sanitizer runtime "
        "(see https://github.com/google/sanitizers/issues/611 for details"
        "). If you want to run %s library under sanitizers please remove "
        "RTLD_DEEPBIND from dlopen flags.\n",
        filename, filename);
    Die();
  }

Yes, if the shared library at filename doesn't exist, I think ASan needn't report and die.

[janwilmans]

The message was added because people we getting super weird false positives. I think the message is very helpful, the Die(), I have removed locally and I just know to that I can expect weirdness if I need this message. Maybe we should just add, ASAN can continue but please don't report issues if you see this message and be aware you can experience false positives from RTLD_DEEPBIND -loaded dynamic libraries.

[janwilmans]

Frankly, your best choice is to contact the vendor for the SDK and request that they support asan in whatever environment this SDK is being used. We've seen cases in the past when a closed-source SDK did something extremely asan-hostile and there is very little we could do there.

In practice this is very hard, some vendors aren't even aware the are doing this themselves, they load libraries, that load libraries etc. We should have some way to allow the user to sanitize their own code at least...

[ojwb]

please remove RTLD_DEEPBIND from dlopen flags

It's intercepting the dlopen() call already, so could asan provide a way to strip out the RTLD_DEEPBIND flag before calling the real dlopen()?

To follow the error message's suggestion and remove the flag it seems one currently has to rebuild the code which uses the flag. In the case I've just hit the problematic dlopen() call is in the PHP interpreter, so that means rebuilding all of PHP in order to use asan on an externally built PHP module. Looking at PHP's code it already avoids using RTLD_DEEPBIND automatically if PHP itself is built with asan or memsan, so if I could tell asan to strip out the flag it ought to work:

# if defined(RTLD_GROUP) && defined(RTLD_WORLD) && defined(RTLD_PARENT)
#  define DL_LOAD(libname)          dlopen(libname, PHP_RTLD_MODE | RTLD_GLOBAL | RTLD_GROUP | RTLD_WORLD | RTLD_PARENT)
# elif defined(RTLD_DEEPBIND) && !defined(__SANITIZE_ADDRESS__) && !__has_feature(memory_sanitizer)
#  define DL_LOAD(libname)          dlopen(libname, PHP_RTLD_MODE | RTLD_GLOBAL | RTLD_DEEPBIND)
# else
#  define DL_LOAD(libname)          dlopen(libname, PHP_RTLD_MODE | RTLD_GLOBAL)
# endif

[janwilmans]

@ojwb RTLD_DEEPBIND is there for a reason, you cannot just remove it and expect it to work ok. Specifically RTLD_DEEPBIND causes symbol lookups to happen in the loaded library and if you remove it, the symbols will be looked up in one of its parents, which is functionally not the same.

[kotee4ko]

as a workaround solution:

colordiff --suppress-common-lines -U16 <(objdump -Mintel -dC /usr/lib/x86_64-linux-gnu/libasan.so.8.0.0) <(objdump -Mintel -dC /tmp/libasan.so.8.0.0)

 0000000000060270 <__interceptor_dlopen.part.0>:
    60270:      55                      push   rbp
    60271:      48 89 e5                mov    rbp,rsp
    60274:      41 57                   push   r15
    60276:      41 56                   push   r14
    60278:      41 55                   push   r13
    6027a:      41 54                   push   r12
    6027c:      41 89 f4                mov    r12d,esi
    6027f:      53                      push   rbx
    60280:      48 89 fb                mov    rbx,rdi
    60283:      48 81 ec 28 08 00 00    sub    rsp,0x828
    6028a:      48 8d 05 0b 83 11 00    lea    rax,[rip+0x11830b]        # 17859c <__asan::asan_inited>
    60291:      8b 30                   mov    esi,DWORD PTR [rax]
    60293:      85 f6                   test   esi,esi
    60295:      0f 84 95 01 00 00       je     60430 <__interceptor_dlopen.part.0+0x1c0>
    6029b:      48 85 db                test   rbx,rbx
    6029e:      74 10                   je     602b0 <__interceptor_dlopen.part.0+0x40>
    602a0:      48 8d 05 19 a0 1a 00    lea    rax,[rip+0x1aa019]        # 20a2c0 <__sanitizer::common_flags_dont_use>
    602a7:      80 b8 f5 00 00 00 00    cmp    BYTE PTR [rax+0xf5],0x0
    602ae:      75 50                   jne    60300 <__interceptor_dlopen.part.0+0x90>
    602b0:      48 8d 05 29 4b 11 00    lea    rax,[rip+0x114b29]        # 174de0 <__asan::asan_flags_dont_use_directly>
    602b7:      80 78 59 00             cmp    BYTE PTR [rax+0x59],0x0
    602bb:      0f 85 5f 01 00 00       jne    60420 <__interceptor_dlopen.part.0+0x1b0>
-   602c1:      44 89 e6                mov    esi,r12d
+   602c1:      31 f6                   xor    esi,esi
+   602c3:      90                      nop
    602c4:      48 89 df                mov    rdi,rbx
    602c7:      e8 14 40 09 00          call   f42e0 <__sanitizer::CheckNoDeepBind(char const*, int)>
    602cc:      44 89 e6                mov    esi,r12d
    602cf:      48 89 df                mov    rdi,rbx
    602d2:      ff 15 98 4f 11 00       call   QWORD PTR [rip+0x114f98]        # 175270 <__interception::real_dlopen>
    602d8:      48 89 c3                mov    rbx,rax
    602db:      e8 40 2e 0a 00          call   103120 <__sanitizer::Symbolizer::GetOrInit()>
    602e0:      48 89 c7                mov    rdi,rax
    602e3:      e8 78 1f 0a 00          call   102260 <__sanitizer::Symbolizer::InvalidateModuleList()>
    602e8:      48 8d 65 d8             lea    rsp,[rbp-0x28]
    602ec:      48 89 d8                mov    rax,rbx
    602ef:      5b                      pop    rbx
    602f0:      41 5c                   pop    r12
    602f2:      41 5d                   pop    r13
    602f4:      41 5e                   pop    r14
    602f6:      41 5f                   pop    r15
    602f8:      5d                      pop    rbp
    602f9:      c3                      ret

Please keep in mind, that this patch could break calls to free() in some rare cases, when called from library which was loaded by dlopen()