Problem with opendir / fstatfs in Firefox on Mac OSX

[GoogleCodeExporter]

I briefly talked to kcc about this issue and he suggested to file a bug, even 
if the bug might be in our code. I'm trying to run Firefox with ASan on Mac OSX 
now and I'm hitting this bug constantly on startup:

https://bugzilla.mozilla.org/show_bug.cgi?id=935795

There are multiple things that could be going wrong, like us bypassing an 
interceptor (we've had that before for certain calls to free), a missing 
interceptor, or just a regular bug that can't be seen from the stack. The bug 
is certainly OSX only, we haven't hit such a problem on Linux before.

Since this is a blocker for us, I'm happy for any debugging suggestions or 
patches to try :)

kcc also mentioned: " The report looks broken. First, the stack is short -- for 
some reason we failed to unwind it fully. Second, it sounds weird -- why would 
opendir touch so many local variables. Does this code perform any kind of stack 
unwinding (similar to throwing exception or longjmp)? "

I am not aware that we have any form of custom unwinding involved here, but I 
could be wrong :)

Original issue reported on code.google.com by decoder...@googlemail.com on 6 Aug 2014 at 12:20

[GoogleCodeExporter]

Given the name of the function I suspect that it actually goes into JavaScript, 
where it may throw a JS exception somehow... 
glider may have more Mac-specific ideas, but he is OOO for a while... 
I would start with two things: 

1. run under gdb, set breakpoint to __asan_report_error, see the failure stack 
trace.
2. Insert printfs into every function between js::ctypes::FunctionType::Call 
and the point of failure (where you can). (Print the current values of SP). 

Original comment by konstant...@gmail.com on 6 Aug 2014 at 12:30

[GoogleCodeExporter]

I tried breaking on __asan_report_error but did not succeed (using either lldb 
or gdb on Mac). It just doesn't find the function (in fact, it doesn't seem to 
find any of the functions that are in the ASan dylib). Any further hints on how 
to get this working in the debugger?

Original comment by decoder...@googlemail.com on 8 Aug 2014 at 1:17

[GoogleCodeExporter]

Does nm show any symbols in the ASan dylib?
Have you tried attaching to a running process and/or setting a breakpoint once 
the program has started?
Looking at the report now.

Original comment by ramosian.glider@gmail.com on 21 Aug 2014 at 5:09

[GoogleCodeExporter]

If you're ok with building a custom Clang, I'd try to insert a Print() or 
Report() call into the fstatfs interceptor to dump the contents of the memory 
before the call to REAL(fstatfs)() (see 
projects/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc)

BTW there's the following warning in the interceptor:

3068   // FIXME: under ASan the call below may write to freed memory and corrupt
3069   // its metadata. See
3070   // https://code.google.com/p/address-sanitizer/issues/detail?id=321.

It shouldn't bother you as your report happens on a stack-allocated memory 
(never saw a stack address like 0x000119101c00 before, but vmmap says it's 
possible)

Original comment by ramosian.glider@gmail.com on 21 Aug 2014 at 5:25

[GoogleCodeExporter]

Is there a log with shadow layout for the given address?
Maybe I'm missing something but I can't see how this could be an underflow.

On an unrelated note, is it possible to make the JS engine generate some unwind 
info (e.g. frame pointers)?

Original comment by ramosian.glider@gmail.com on 21 Aug 2014 at 5:29

[GoogleCodeExporter]

Note that in the following report:

> ==54944==ERROR: AddressSanitizer: stack-buffer-underflow on address 
0x000119101c00 at pc 0x10002328d bp 0x1191018a0 sp 0x119101880
> WRITE of size 2168 at 0x000119101c00 thread T19
>     #0 0x10002328c in wrap_fstatfs 
(/Users/jruderman/llvm/build/Release/lib/clang/3.4/lib/darwin/libclang_rt.asan_o
sx_dynamic.dylib+0x1528c)
>     #1 0x7fff89ffcff4 in __opendir2 
(/usr/lib/system/libsystem_c.dylib+0xa0ff4)
>     #2 0x10b43787b in ffi_call_unix64 
(/Users/jruderman/builds/mozilla-central-asan-opt/dist/Nightly.app/Contents/MacO
S/XUL+0x71ce87b)
>     #3 0x60200004a7af
> Address 0x000119101c00 is located in stack of thread T19 at offset 0 in frame
>     #0 0x10b40413f in js::ctypes::FunctionType::Call(JSContext*, unsigned 
int, JS::Value*) 
(/Users/jruderman/builds/mozilla-central-asan-opt/dist/Nightly.app/Contents/MacO
S/XUL+0x719b13f)
>   This frame has 6 object(s):
>     [32, 40) 'obj.i' <== Memory access at offset 0 partially underflows this 
variable
>     [96, 104) 'objTypeProto.i' <== Memory access at offset 0 partially 
underflows this variable
>     [160, 312) 'values' <== Memory access at offset 0 partially underflows 
this variable
>     [352, 504) 'strings' <== Memory access at offset 0 partially underflows 
this variable
>     [544, 568) 'autoCallback' <== Memory access at offset 0 partially 
underflows this variable
>     [608, 616) 'returnType' <== Memory access at offset 0 partially 
underflows this variable
> HINT: this may be a false positive if your program uses some custom stack 
unwind mechanism or swapcontext
>       (longjmp and C++ exceptions *are* supported)

the pc of frame #3 is in the heap.
If fast unwinder was used it would've definitely give up unwinding at this 
point.
Not sure about the slow unwinder, probably there're no checks for stack top and 
bottom there.

Original comment by ramosian.glider@gmail.com on 21 Aug 2014 at 5:56

• Added labels: OpSys-OSX

[GoogleCodeExporter]

Two more questions:
 - is this bug reproducible in a nightly build? I.e. can I download a prebuilt binary somewhere?
 - how does the ASAN_OPTIONS=verbosity=1 output look like? I'm particularly interested in the thread stack ranges.

Original comment by ramosian.glider@gmail.com on 21 Aug 2014 at 6:00

[GoogleCodeExporter]

Here's the output with verbosity=1 (45 MB log, so I uploaded it):

https://users.own-hero.net/~decoder/asan.txt 

I will try to package the build for you (I recently wrote experimental code for 
the dylib stuff that unbreaks packaging) and upload it.

Regarding this question:

> On an unrelated note, is it possible to make the JS engine generate some 
unwind info (e.g. frame pointers)?

how would I do that? Compile with frame pointers enabled? 

Original comment by decoder...@googlemail.com on 21 Aug 2014 at 7:51

[GoogleCodeExporter]

> how would I do that? Compile with frame pointers enabled?
Nope, I was talking about frame pointers in the JITted code generated by 
SpiderMonkey. As frame #3 is in the heap I'm assuming this is JIT-generated 
code and for some reason (probably the missing frame pointers) ASan could not 
unwind past that address.
V8 emits frame pointers by default, but I don't know if SpiderMonkey does.

Original comment by ramosian.glider@gmail.com on 22 Aug 2014 at 7:51

[GoogleCodeExporter]

I've spent some time today checking whether anything could be wrong with 
libffi+ASan.
My hypothesis was that ffi_call() could be somehow messing up with the stack 
frame.
I couldn't prove that on my Linux box, however.

Original comment by ramosian.glider@gmail.com on 22 Aug 2014 at 10:46

[GoogleCodeExporter]

I've talked to our JS developers and they say they don't maintain a frame 
pointer. It would be possible to do so, but they would have to implement that.

Regarding your experiments with Linux: This bug is very likely Mac only. We are 
running a lot of tests and fuzzing on Linux and never hit any problems with 
ffi_call, esp. not this problem. On Mac, I can reproduce it on startup 100% of 
the time. So it's either Mac specific code in our code, or in ASan code.

I will package the browser for you next week (I'm arriving at home Wednesday 
most likely). Then I would also be able to make builds with a custom Clang if 
that's necessary.

Original comment by decoder...@googlemail.com on 22 Aug 2014 at 4:23

[GoogleCodeExporter]

I just uploaded my build to 
https://users.own-hero.net/~decoder/firefox-34.0a1.en-US.mac.dmg

It's a debug+opt build, if you need a pure debug build, let me know and I'll 
make that as well. Let me also know if the issue reproduces for you in that 
build :)

Original comment by decoder...@googlemail.com on 27 Aug 2014 at 4:27

[GoogleCodeExporter]

This task was swapped out by others and I didn't touch it for quite a while :(
But for the record I had managed to reproduce the problem, though hadn't found 
any clues.

Original comment by ramosian.glider@gmail.com on 10 Nov 2014 at 2:31