jacereda / fsatrace

Filesystem access tracer
ISC License
78 stars 12 forks source link

segfault in emiting op #42

Open woody77 opened 2 years ago

woody77 commented 2 years ago

Hello, I'm seeing a segfault in the traced app, when it's trying to write to the shared memory buffer back to fsatrace:

<segv>
#4  emitOp (oc=oc@entry=114, op1=<optimized out>, p2=p2@entry=0x0) at src/emit.c:118
#5  0x00007f6fa63525f3 in fdemit (c=c@entry=114, fd=fd@entry=16) at src/unix/fsatraceso.c:118
#6  0x00007f6fa6352937 in openat64 (fd=-100, p=<optimized out>, f=<optimized out>, m=<optimized out>) at src/unix/fsatraceso.c:269

I don't have much more info at this time, but from looking at the source, is this likely to be running past the end of the buffer?

From my reading of main(), all accesses are buffered in the shared memory buffer until the process is complete, and then written, correct? (no concurrent access) https://github.com/jacereda/fsatrace/blob/master/src/fsatrace.c#L193-L203

And the default logsize is 1MB of text? https://github.com/jacereda/fsatrace/blob/master/src/fsatrace.h#L4

Which can be overridden by setting the env var FSAT_BUF_SIZE?

woody77 commented 2 years ago

I see this even if I set the buffer to larger and larger sizes, up to a size large enough that fsatrace fails to allocate the buffer.

jacereda commented 2 years ago

Hi, thanks for the report.

Can you reproduce this? Can you share how?

From my reading of main(), all accesses are buffered in the shared memory buffer until the process is complete, and then written, correct? (no concurrent access) https://github.com/jacereda/fsatrace/blob/master/src/fsatrace.c#L193-L203

Right.

And the default logsize is 1MB of text? https://github.com/jacereda/fsatrace/blob/master/src/fsatrace.h#L4

Which can be overridden by setting the env var FSAT_BUF_SIZE?

Right.

woody77 commented 2 years ago

The issue that I linked to this: https://github.com/dart-lang/sdk/issues/48413 has the setup where we found it.

It's a pretty deeply nested conglomeration of processes to setup the issue. I haven't been able to reproduce it outside of that context.

jacereda commented 2 years ago

Can you generate a coredump? Also, the buffer is stack-allocated, can you replace that by a malloc() and increase the FSAT_BUF_SIZE?

jacereda commented 2 years ago

Also can you try the current master? That one gets rid of FSAT_BUF_SIZE and has a static 16MB buffer.

woody77 commented 2 years ago

@fangism - I don't know how to update and rebuild fsatrace within the Fuchsia build, can you help with this? (if you push a new binary to CIPD, I can snag it locally and test on my workstation that's seeing this issue).

woody77 commented 2 years ago

Looks like we're getting different results, now:

fsatrace(87243): error: freeing shared memory (4)
charmoniumQ commented 11 months ago

I found this error in 41b02a0 (2020-10-10), which is currently packaged in Nixpkgs, but it seems to work in the latest commit c031f8da (2022-03-30).