search

lsalamon / address-sanitizer

Automatically exported from code.google.com/p/address-sanitizer
0 stars 1 forks source link

CHECK failed (assert() tls + stack) on Linux in 32bit threaded #361

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Build clang+compiler-rt 3.5
2. Use it with attached test file to produce a 32-bit binary multi-threaded
3. AddressSanitizer CHECK failed (sanitizer_linux_libcdep.cc:433)

What is the expected output? What do you see instead?

$ clang -m32 -fsanitize=address test_thread.c -DTHREAD_USE
The program should launch and print the function name instead :
$ ./a.out

==19661==AddressSanitizer CHECK failed: 
/tmp/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:4
33 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0xf638f030, 
0xf638f000)
    #0 0x80da2ad (/tmp/a.out+0x80da2ad)
    #1 0x80de75f (/tmp/a.out+0x80de75f)
    #2 0x80ed8f7 (/tmp/a.out+0x80ed8f7)
    #3 0x80dbe61 (/tmp/a.out+0x80dbe61)
    #4 0x80dc1f2 (/tmp/a.out+0x80dc1f2)
    #5 0x80b4afd (/tmp/a.out+0x80b4afd)
    #6 0xf76f8953 (/lib/i386-linux-gnu/libpthread.so.0+0x5953)

But in 32-bit without thread it works :

$ clang -m32 -fsanitize=address test_thread.c
$ ./a.out
threadfunc

And in 64-bit it works in every case :

clang -m64 -fsanitize=address test_thread.c -DTHREAD_USE
clang -m64 -fsanitize=address test_thread.c

What version of the product are you using? On what operating system?

Clang 3.5  (also tested on 3.4 same result)

Host for compiling the test and running it :
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.2 (wheezy)
Release:    7.2
Codename:   wheezy

Please provide any additional information below.

I used another system to build clang+asan :
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.10 (squeeze)
Release:    6.0.10
Codename:   squeeze

But even if I build the test file on this machine the resulting binary doesn't 
work on 'wheezy'

I also tried changing the stack value (ulimit (-H/-S) -s) without success.

Informations from the libc (in case that would be helpful) :

/lib/i386-linux-gnu/libc-2.13.so 
GNU C Library (Debian EGLIBC 2.13-38) stable release version 2.13, by Roland 
McGrath et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.4.7.
Compiled on a Linux 3.2.35 system on 2012-12-30.
Available extensions:
    crypt add-on version 2.1 by Michael Glad and others
    GNU Libidn by Simon Josefsson
    Native POSIX Threads Library by Ulrich Drepper et al
    BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.

Original issue reported on code.google.com by jon34056...@gmail.com on 4 Dec 2014 at 1:57

Attachments:

GoogleCodeExporter commented 9 years ago 
Quick update this is also reproducible on Wheezy 7.7.

But with Jessie (Debian 8) it seems to be fixed as I'm not able to reproduce it.

Original comment by jon34056...@gmail.com on 30 Dec 2014 at 9:02

GoogleCodeExporter commented 9 years ago 
Can you please rebuild the program with -g and:
1. Run the program with ASAN_OPTIONS=verbosity=2 and attach the output
2. Post a symbolized ASan report 
(http://clang.llvm.org/docs/AddressSanitizer.html)
3. Run with ASAN_OPTIONS=sleep_before_dying=100, attach gdb and post the 
stacktrace?

Original comment by ramosian.glider@gmail.com on 30 Dec 2014 at 9:40

GoogleCodeExporter commented 9 years ago 
Thanks for your answer.

Here you go (with clang 3.5.1 225002):

1. Verbose output symbolized (so it's also 2.)

==3725==Parsed ASAN_OPTIONS: verbosity=2
==3725==AddressSanitizer: failed to intercept '__isoc99_printf'
==3725==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_snprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_fprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
==3725==AddressSanitizer: libc interceptors initialized
|| `[0x40000000, 0xffffffff]` || HighMem    ||
|| `[0x28000000, 0x3fffffff]` || HighShadow ||
|| `[0x24000000, 0x27ffffff]` || ShadowGap  ||
|| `[0x20000000, 0x23ffffff]` || LowShadow  ||
|| `[0x00000000, 0x1fffffff]` || LowMem     ||
MemToShadow(shadow): 0x24000000 0x247fffff 0x25000000 0x27ffffff
redzone=16
max_redzone=2048
quarantine_size=64M
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 20000000
==3725==Installed the sigaction for signal 11
==3725==SetCurrentThread: 0xf770e000 for thread 0xf753b6f0
==3725==T0: stack [0xff6c0000,0xffec0000) size 0x800000; local=0xffebdaec
==3725==AddressSanitizer Init done
==3725==SetCurrentThread: 0xf770e078 for thread 0xf5fffb70
==3725==AddressSanitizer CHECK failed: 
/tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.
cc:433 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0xf6000030, 
0xf6000000)
    #0 0x80f2b86 in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:70
    #1 0x80f83bc in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:76
    #2 0x8108ef6 in __sanitizer::GetThreadStackAndTls(bool, unsigned long*, unsigned long*, unsigned long*, unsigned long*) /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:433
    #3 0x80f57d4 in __asan::AsanThread::SetThreadStackAndTls() /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:187
    #4 0x80f57d4 in __asan::AsanThread::Init() /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:146
    #5 0x80f5ac9 in __asan::AsanThread::ThreadStart(unsigned long) /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:159
    #6 0x807a8c2 in asan_thread_start(void*) /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:170
    #7 0xf76e1953 in start_thread (/lib/i386-linux-gnu/libpthread.so.0+0x5953)

3. Stacktrace with gdb

0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt
#0  0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
#1  0xf75f30c2 in sleep () from /lib/i386-linux-gnu/libc.so.6
#2  0x0810c3ff in main ()
(gdb) info threads
  Id   Target Id         Frame 
  2    Thread 0xf5fffb70 (LWP 3814) "a.out" 0xf75f329c in nanosleep ()
   from /lib/i386-linux-gnu/libc.so.6
* 1    Thread 0xf75526f0 (LWP 3813) "a.out" 0xf75f329c in nanosleep ()
   from /lib/i386-linux-gnu/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0xf5fffb70 (LWP 3814))]
#0  0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt
#0  0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
#1  0xf75f30c2 in sleep () from /lib/i386-linux-gnu/libc.so.6
#2  0x0810958b in __sanitizer::SleepForSeconds (seconds=100)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:72
#3  0x080f3450 in __asan::AsanDie ()
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:46
#4  0x080f832b in __sanitizer::Die ()
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:63
#5  0x080f2c24 in __asan::AsanCheckFailed (
    file=0x811a70c "/tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc", line=433, 
    cond=0x811a874 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))", 
    v1=4127195184, v2=4127195136)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:71
#6  0x080f83bd in __sanitizer::CheckFailed (
    file=0x811a70c "/tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc", line=433, 
    cond=0x811a874 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))", 
    v1=4127195184, v2=4127195136)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:76
#7  0x08108ef7 in __sanitizer::GetThreadStackAndTls (main=false, 
    stk_addr=0xf7387010, stk_size=0xf7387014, tls_addr=0xf7387018, 
    tls_size=0xf5fff2e8)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:433
#8  0x080f57d5 in SetThreadStackAndTls (this=0xf7387000)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:187
#9  __asan::AsanThread::Init (this=this@entry=0xf7387000)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:146
#10 0x080f5aca in __asan::AsanThread::ThreadStart (this=0xf7387000, os_id=3814)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:159
#11 0x0807a8c3 in asan_thread_start (arg=0xf7387000)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:170
#12 0xf76f8954 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#13 0xf762895e in clone () from /lib/i386-linux-gnu/libc.so.6

On a side note even though it's specified to sleep for 100 seconds, I had to 
increase the sleep() time in the test_thread.c code otherwise it exited after 
the specified sleep time in the code (so after 1 second).
Thanks again for your help.

Original comment by jon34056...@gmail.com on 30 Dec 2014 at 9:09

GoogleCodeExporter commented 9 years ago 
FYI this now also happens on our buildbot:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/15213/steps/run%
20sanitizer%20tests%20in%20gcc%20build/logs/stdio

This one is interesting:
[ RUN      ] SanitizerLinux.ThreadDescriptorSize
/mnt/b/sanitizer-buildbot1/sanitizer-x86_64-linux/build/llvm/projects/compiler-r
t/lib/sanitizer_common/tests/sanitizer_linux_test.cc:230: Failure
Value of: ThreadDescriptorSize()
  Actual: 1216
Expected: (uptr)result
Which is: 1168
[  FAILED  ] SanitizerLinux.ThreadDescriptorSize (4 ms)

I've checked that versions of libc and libpthread are 2.13, but the descriptor 
size we calculate seems to be coincide with size used in older versions.

Original comment by samso...@google.com on 9 Jan 2015 at 11:00

GoogleCodeExporter commented 9 years ago 
FYI this now also happens on our buildbot:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/15213/steps/run%
20sanitizer%20tests%20in%20gcc%20build/logs/stdio

This one is interesting:
[ RUN      ] SanitizerLinux.ThreadDescriptorSize
/mnt/b/sanitizer-buildbot1/sanitizer-x86_64-linux/build/llvm/projects/compiler-r
t/lib/sanitizer_common/tests/sanitizer_linux_test.cc:230: Failure
Value of: ThreadDescriptorSize()
  Actual: 1216
Expected: (uptr)result
Which is: 1168
[  FAILED  ] SanitizerLinux.ThreadDescriptorSize (4 ms)

I've checked that versions of libc and libpthread are 2.13, but the descriptor 
size we calculate seems to be coincide with size used in older versions.

Original comment by samso...@google.com on 9 Jan 2015 at 11:00

GoogleCodeExporter commented 9 years ago 
Hmm.

Here the actual size is 1168, but the one we expect is 1216. Originally, we 
expected 1168 on 2.13 and lower, exactly because someone had reported 1168 on 
2.13 and 2.11. This was changed after Jakub gathered descriptor sizes for 
different glibc versions:

https://gcc.gnu.org/ml/gcc-patches/2013-12/msg00287.html

Here we have 1216 for 2.13. I guess no one noticed at the time that Jakub's 
patch changed the value for 2.13.

The question is, is it a problem with Jakub's data (unlikely because it was 
gathered programmatically) or if there's another variable (besides the version 
number) that can affect the descriptor size.

Original comment by earth...@google.com on 19 Jan 2015 at 3:39

GoogleCodeExporter commented 9 years ago 
FTR, I downloaded 2.13 from https://packages.debian.org/wheezy/libc6 and used 
Jakub's method. I got FIRST_32_SECOND_64(1216, 2304) - the same values we 
originally used.

Jakub's binaries were from RedHat, so I wonder if it's different there?

As a temporary solution I'm going to reinstate original values for 2.13, to 
make our bots green at least.

Original comment by earth...@chromium.org on 23 Jan 2015 at 9:08

GoogleCodeExporter commented 9 years ago 
> I got FIRST_32_SECOND_64(1216, 2304)

Typo - should be (1168, 2304)

Original comment by earth...@google.com on 23 Jan 2015 at 9:11

GoogleCodeExporter commented 9 years ago 
Committed r226938

Original comment by earth...@google.com on 23 Jan 2015 at 9:14

GoogleCodeExporter commented 9 years ago 
I just tried again with Clang 3.6.1 (clang version 3.6.1 (branches/release_36 
232166)

And I can still reproduce this issue.
Do you have any tip so I could try to solve this issue ?

Original comment by jon34056...@gmail.com on 13 Mar 2015 at 4:20

GoogleCodeExporter commented 9 years ago 
Adding Project:AddressSanitizer as part of GitHub migration.

Original comment by ramosian.glider@gmail.com on 30 Jul 2015 at 9:06