Intermittent failure in `mutex_pi_stress`

[Keno]

Just saw this on Anubis:

[FATAL /data/keno/rr/src/RecordSession.cc:1506:process_syscall_entry() errno: SUCCESS]
 (task 50829 (rec:50829) at time 20260)
 -> Assertion `t->desched_rec() || is_rrcall_notify_syscall_hook_exit_syscall( t->regs().original_syscallno(), t->arch()) || t->ip() == t->vm() ->privileged_traced_syscall_ip() .increment_by_syscall_insn_length(t->arch())' failed to hold.
[FATAL /data/keno/rr/src/log.cc:289:emergency_debug() errno: SUCCESS] (session doesn't look interactive, aborting emergency debugging)

Seems like we also saw this on the KNL buildbot: https://ci.rr-project.org/#/builders/4/builds/93/steps/9/logs/stdio

[FATAL /home/rrbuildbot/rr-knl-baremetal/build/rr/src/RecordSession.cc:1508:process_syscall_entry() errno: SUCCESS]
 (task 228422 (rec:228422) at time 46702)
 -> Assertion `t->desched_rec() || is_rrcall_notify_syscall_hook_exit_syscall( t->regs().original_syscallno(), t->arch()) || t->ip() == t->vm() ->privileged_traced_syscall_ip() .increment_by_syscall_insn_length(t->arch())' failed to hold.
[FATAL /home/rrbuildbot/rr-knl-baremetal/build/rr/src/log.cc:289:emergency_debug() errno: SUCCESS] (session doesn't look interactive, aborting emergency debugging)

[rocallahan]

Not easily reproduced locally :-/.

[rocallahan]

20 concurrent test runs on Anubis don't fail either.

[Keno]

Try running the whole test suite with -j40. I'm seeing intermittent s fairly often

On Feb 15, 2017 5:51 PM, "rocallahan" notifications@github.com wrote:

20 concurrent test runs on Anubis don't fail either.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mozilla/rr/issues/1984#issuecomment-280166642, or mute the thread https://github.com/notifications/unsubscribe-auth/ABO1l4dtw_rGSnwxLsCFK6EPiERzOTr9ks5rc4FHgaJpZM4MCR8X .

[rocallahan]

OK!

[jiridanek]

This assertion happened to me in production usage, as I was running rr 5.4.0 in a Fedora 34 Beta Docker container. I was trying to reproduce my issue https://issues.apache.org/jira/browse/DISPATCH-782.

I started the Docker with --privileged flag. I was running rr record --chaos in a loop, when this happened. First few initial lines is output from the Dispatch binary running under rr.

2021-04-02 12:06:37.289615 +0000 AGENT (info) Activating management agent on $_management_internal
2021-04-02 12:06:37.291084 +0000 ROUTER_CORE (info) In-process subscription L/$management
2021-04-02 12:06:37.291305 +0000 ROUTER_CORE (info) In-process subscription L/$_management_internal
2021-04-02 12:06:37.294804 +0000 POLICY (info) Policy configured maxConnections: 65535, policyDir: '',access rules enabled: 'false', use hostname patterns: 'false'
2021-04-02 12:06:37.295268 +0000 AGENT (debug) Add entity: PolicyEntity(defaultVhost=$default, enableVhostNamePatterns=False, enableVhostPolicy=False, identity=policy/None, maxConnections=65535, maxMessageSize=0, name=policy/None, policyDir=, type=org.apache.qpid.dispatch.policy)
[FATAL /builddir/build/BUILD/rr-5.4.0/src/RecordSession.cc:1807:process_syscall_entry()] 
 (task 1013 (rec:1013) at time 5450)
 -> Assertion `t->desched_rec() || is_rrcall_notify_syscall_hook_exit_syscall( t->regs().original_syscallno(), t->arch()) || t->ip() == t->vm() ->privileged_traced_syscall_ip() .increment_by_syscall_insn_length(t->arch())' failed to hold. Stashed signal pending on syscall entry when it shouldn't be: {signo:SIGSTKFLT,errno:SUCCESS,code:sicode(1)}; IP=0x7fda9f99ec2e
Tail of trace dump:
{
  real_time:4229827.211682 global_time:5430, event:`SYSCALL: read' (state:ENTERING_SYSCALL) tid:1013, ticks:56791756
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x237f480 rdi:0x9 rbp:0x9 rsp:0x681ffd40 r8:0x0 r9:0x27 r10:0x46ba28 r11:0x246 r12:0x237f480 r13:0x1 r14:0x9 r15:0x20f75e0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x0 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.211722 global_time:5431, event:`SYSCALL: read' (state:EXITING_SYSCALL) tid:1013, ticks:56791756
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x237f480 rdi:0x9 rbp:0x9 rsp:0x681ffd40 r8:0x0 r9:0x27 r10:0x46ba28 r11:0x246 r12:0x237f480 r13:0x1 r14:0x9 r15:0x20f75e0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x0 fs_base:0x5cb048fb2c0 gs_base:0x0
  { tid:1013, addr:0x237f480, length:(nil) }
}
{
  real_time:4229827.212416 global_time:5432, event:`SYSCALLBUF_FLUSH' tid:1013, ticks:57062780
  { syscall:'close', ret:0x0, size:0x10 }
  { syscall:'gettimeofday', ret:0x0, size:0x20 }
}
{
  real_time:4229827.212422 global_time:5433, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1013, ticks:57062780
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7fff87ba2610 rsi:0x7fda9fa3ce40 rdi:0xffffff9c rbp:0x7fda9fa3ce40 rsp:0x7fff87ba2578 r8:0x20f3520 r9:0x43 r10:0x0 r11:0x246 r12:0x0 r13:0x1 r14:0x223c180 r15:0x3fa5a0 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.212425 global_time:5434, event:`SYSCALLBUF_RESET' tid:1013, ticks:57062780
}
{
  real_time:4229827.212479 global_time:5435, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1013, ticks:57062780
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7fff87ba2610 rsi:0x7fda9fa3ce40 rdi:0xffffff9c rbp:0x7fda9fa3ce40 rsp:0x7fff87ba2578 r8:0x20f3520 r9:0x43 r10:0x0 r11:0x246 r12:0x0 r13:0x1 r14:0x223c180 r15:0x3fa5a0 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
  { tid:1013, addr:0x7fff87ba2610, length:0x90 }
}
{
  real_time:4229827.212542 global_time:5436, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:1013, ticks:57063559
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xa4 rsi:0x7fff87ba3030 rdi:0x2 rbp:0x7fda9f791000 rsp:0x681ffdf0 r8:0x0 r9:0x7fff87ba2660 r10:0x7fda9fa353e0 r11:0x246 r12:0x2 r13:0x7fda9fa72440 r14:0x7fff87ba3030 r15:0xa4 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.212585 global_time:5437, event:`SYSCALL: write' (state:EXITING_SYSCALL) tid:1013, ticks:57063559
rax:0xa4 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0xa4 rsi:0x7fff87ba3030 rdi:0x2 rbp:0x7fda9f791000 rsp:0x681ffdf0 r8:0x0 r9:0x7fff87ba2660 r10:0x7fda9fa353e0 r11:0x246 r12:0x2 r13:0x7fda9fa72440 r14:0x7fff87ba3030 r15:0xa4 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.212723 global_time:5438, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1013, ticks:57082195
rax:0xffffffffffffffda rbx:0x7fff87ba32a0 rcx:0xffffffffffffffff rdx:0x7fff87ba31b0 rsi:0x4720310b1a0 rdi:0xffffff9c rbp:0xffffff9c rsp:0x7fff87ba31a8 r8:0x0 r9:0x7fff87367f00 r10:0x0 r11:0x246 r12:0x7fff87ba31b0 r13:0x34583d760770 r14:0x20f75e0 r15:0x2280a00 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.212780 global_time:5439, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1013, ticks:57082195
rax:0x0 rbx:0x7fff87ba32a0 rcx:0xffffffffffffffff rdx:0x7fff87ba31b0 rsi:0x4720310b1a0 rdi:0xffffff9c rbp:0xffffff9c rsp:0x7fff87ba31a8 r8:0x0 r9:0x7fff87367f00 r10:0x0 r11:0x246 r12:0x7fff87ba31b0 r13:0x34583d760770 r14:0x20f75e0 r15:0x2280a00 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
  { tid:1013, addr:0x7fff87ba31b0, length:0x90 }
}
{
  real_time:4229827.212875 global_time:5440, event:`SYSCALLBUF_FLUSH' tid:1013, ticks:57087685
  { syscall:'gettimeofday', ret:0x0, size:0x20 }
}
{
  real_time:4229827.212883 global_time:5441, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1013, ticks:57087685
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7fff87ba2680 rsi:0x7fda9fa3ce40 rdi:0xffffff9c rbp:0x7fda9fa3ce40 rsp:0x7fff87ba25e8 r8:0x20f3520 r9:0x43 r10:0x0 r11:0x246 r12:0x0 r13:0x1 r14:0x21b5850 r15:0x3fa588 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.212886 global_time:5442, event:`SYSCALLBUF_RESET' tid:1013, ticks:57087685
}
{
  real_time:4229827.212935 global_time:5443, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1013, ticks:57087685
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7fff87ba2680 rsi:0x7fda9fa3ce40 rdi:0xffffff9c rbp:0x7fda9fa3ce40 rsp:0x7fff87ba25e8 r8:0x20f3520 r9:0x43 r10:0x0 r11:0x246 r12:0x0 r13:0x1 r14:0x21b5850 r15:0x3fa588 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
  { tid:1013, addr:0x7fff87ba2680, length:0x90 }
}
{
  real_time:4229827.212991 global_time:5444, event:`SYSCALL: write' (state:ENTERING_SYSCALL) tid:1013, ticks:57088468
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x119 rsi:0x7fff87ba30a0 rdi:0x2 rbp:0x7fda9f791000 rsp:0x681ffdf0 r8:0x0 r9:0x7fff87ba26d0 r10:0x7fda9fa353e0 r11:0x246 r12:0x2 r13:0x7fda9fa72440 r14:0x7fff87ba30a0 r15:0x119 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.213038 global_time:5445, event:`SYSCALL: write' (state:EXITING_SYSCALL) tid:1013, ticks:57088468
rax:0x119 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x119 rsi:0x7fff87ba30a0 rdi:0x2 rbp:0x7fda9f791000 rsp:0x681ffdf0 r8:0x0 r9:0x7fff87ba26d0 r10:0x7fda9fa353e0 r11:0x246 r12:0x2 r13:0x7fda9fa72440 r14:0x7fff87ba30a0 r15:0x119 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x1 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.213578 global_time:5446, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1013, ticks:57301025
rax:0xffffffffffffffda rbx:0x7fff87ba3020 rcx:0xffffffffffffffff rdx:0x7fff87ba2f30 rsi:0x4720310b210 rdi:0xffffff9c rbp:0xffffff9c rsp:0x7fff87ba2f28 r8:0x0 r9:0x7fff87367f00 r10:0x0 r11:0x246 r12:0x7fff87ba2f30 r13:0x34583d760770 r14:0x20f75e0 r15:0x2288400 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.213631 global_time:5447, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1013, ticks:57301025
rax:0x0 rbx:0x7fff87ba3020 rcx:0xffffffffffffffff rdx:0x7fff87ba2f30 rsi:0x4720310b210 rdi:0xffffff9c rbp:0xffffff9c rsp:0x7fff87ba2f28 r8:0x0 r9:0x7fff87367f00 r10:0x0 r11:0x246 r12:0x7fff87ba2f30 r13:0x34583d760770 r14:0x20f75e0 r15:0x2288400 rip:0x7fda9f99ec2e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x5cb048fb2c0 gs_base:0x0
  { tid:1013, addr:0x7fff87ba2f30, length:0x90 }
}
{
  real_time:4229827.213704 global_time:5448, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:1013, ticks:57302481
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x80000 rsi:0x4720310b210 rdi:0xffffff9c rbp:0x7fda9f791000 rsp:0x681ffdf0 r8:0x0 r9:0x7fff8737e660 r10:0x0 r11:0x246 r12:0x80000 r13:0xffffff9c r14:0x0 r15:0x4720310b210 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x5cb048fb2c0 gs_base:0x0
}
{
  real_time:4229827.213872 global_time:5449, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:1013, ticks:57302481
rax:0x9 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x80000 rsi:0x4720310b210 rdi:0xffffff9c rbp:0x7fda9f791000 rsp:0x681ffdf0 r8:0x0 r9:0x7fff8737e660 r10:0x0 r11:0x246 r12:0x80000 r13:0xffffff9c r14:0x0 r15:0x4720310b210 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x5cb048fb2c0 gs_base:0x0
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x5a)[0x55d179254e2a]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x4b5)[0x55d179154035]
rr(+0xa2b9f)[0x55d179161b9f]
rr(+0xa2d34)[0x55d179161d34]
rr(_ZN2rr13RecordSession21process_syscall_entryEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultENS_13SupportedArchE+0x4a4)[0x55d17918b424]
rr(_ZN2rr13RecordSession29handle_seccomp_traced_syscallEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x511)[0x55d1791821c1]
rr(_ZN2rr13RecordSession19handle_ptrace_eventEPPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x551)[0x55d179183311]
rr(_ZN2rr13RecordSession11record_stepEv+0x407)[0x55d17918ffd7]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x1111)[0x55d179180751]
rr(main+0x138)[0x55d1790fab88]
/lib64/libc.so.6(__libc_start_main+0xd5)[0x7f80c54cdb75]
rr(_start+0x2e)[0x55d1790fd77e]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:1013' /main/qpid-dispatch/build/tests/unit_tests

[jiridanek]

The issue can be reproduced very easily (too easily, rr crashes way too easy, so I cannot use rr to look for crashes in my test).

[khuey]

Can you reproduce this on rr tip? I haven't been able to reproduce this (while following the steps in https://issues.apache.org/jira/browse/DISPATCH-782).

[jiridanek]

I can't, git tip does not crash for me. I let the loop run for an hour already and no crashes so far. The Fedora 34 rr package would already surely crash in all that time. I looked at the Fedora spec file and I saw nothing wildly different from how I compiled, https://src.fedoraproject.org/rpms/rr/blob/rawhide/f/rr.spec.

[jiridanek]

@khuey Hi, may I ask for a rr release in order to get a version where this crash does not happen? My colleagues using Fedora are on the latest 5.5.0 version of rr which suffers from this. Having to build custom binary of rr from source is a significant inconvenience. Thanks.

[khuey]

5.5.0 was released several months after you said you couldn't reproduce this: https://github.com/rr-debugger/rr/releases/tag/5.5.0

So if your colleagues are still seeing it it sounds like it wasn't fixed.

[jiridanek]

🤦🏽 looks that way, given the way 5.5.0 seems to be crashing. I will try to look for a reproducer for myself

[FATAL /builddir/build/BUILD/rr-5.5.0/src/RecordSession.cc:1840:process_syscall_entry()] 
 (task 1221895 (rec:1221895) at time 724012)
 -> Assertion `t->desched_rec() || is_rrcall_notify_syscall_hook_exit_syscall( t->regs().original_syscallno(), t->arch()) || t->ip() == t->vm() ->privileged_traced_syscall_ip() .increment_by_syscall_insn_length(t->arch())' failed to hold. Stashed signal pending on syscall entry when it shouldn't be: {signo:SIGSTKFLT,errno:SUCCESS,code:sicode(1)}; IP=0x7f8209c83a59
Tail of trace dump:
{
  real_time:646861.545619 global_time:723992, event:`SYSCALLBUF_FLUSH' tid:1221895, ticks:81743569
  { syscall:'write', ret:0xcd, size:0x10, desched:1 }
  { syscall:'gettimeofday', ret:0x0, size:0x20 }
}
{
  real_time:646861.545623 global_time:723993, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1221895, ticks:81743569
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbd0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb38 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
}
{
  real_time:646861.545625 global_time:723994, event:`SYSCALLBUF_RESET' tid:1221895, ticks:81743569
}
{
  real_time:646861.545648 global_time:723995, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1221895, ticks:81743569
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbd0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb38 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
  { tid:1221895, addr:0x7f81f8b5cbd0, length:0x90 }
}
{
  real_time:646861.545694 global_time:723996, event:`SYSCALLBUF_FLUSH' tid:1221895, ticks:81745323
  { syscall:'write', ret:0xd8, size:0x10, desched:1 }
  { syscall:'gettimeofday', ret:0x0, size:0x20 }
}
{
  real_time:646861.545697 global_time:723997, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1221895, ticks:81745323
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbe0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb48 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
}
{
  real_time:646861.545698 global_time:723998, event:`SYSCALLBUF_RESET' tid:1221895, ticks:81745323
}
{
  real_time:646861.545717 global_time:723999, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1221895, ticks:81745323
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbe0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb48 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
  { tid:1221895, addr:0x7f81f8b5cbe0, length:0x90 }
}
{
  real_time:646861.545755 global_time:724000, event:`SYSCALLBUF_FLUSH' tid:1221895, ticks:81747106
  { syscall:'write', ret:0xc6, size:0x10, desched:1 }
  { syscall:'gettimeofday', ret:0x0, size:0x20 }
}
{
  real_time:646861.545758 global_time:724001, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1221895, ticks:81747106
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbd0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb38 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
}
{
  real_time:646861.545759 global_time:724002, event:`SYSCALLBUF_RESET' tid:1221895, ticks:81747106
}
{
  real_time:646861.545777 global_time:724003, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1221895, ticks:81747106
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbd0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb38 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
  { tid:1221895, addr:0x7f81f8b5cbd0, length:0x90 }
}
{
  real_time:646861.545812 global_time:724004, event:`SYSCALLBUF_FLUSH' tid:1221895, ticks:81748859
  { syscall:'write', ret:0xd8, size:0x10, desched:1 }
  { syscall:'gettimeofday', ret:0x0, size:0x20 }
}
{
  real_time:646861.545815 global_time:724005, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:1221895, ticks:81748859
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbe0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb48 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
}
{
  real_time:646861.545817 global_time:724006, event:`SYSCALLBUF_RESET' tid:1221895, ticks:81748859
}
{
  real_time:646861.545835 global_time:724007, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:1221895, ticks:81748859
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x7f81f8b5cbe0 rsi:0x7f8209db36f0 rdi:0xffffff9c rbp:0x0 rsp:0x7f81f8b5cb48 r8:0x7f81e4069f20 r9:0x73 r10:0x0 r11:0x246 r12:0x7f8209db36f0 r13:0x1 r14:0x142ff00 r15:0x4c3308 rip:0x7f8209cfb4ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f81f8b5f640 gs_base:0x0
  { tid:1221895, addr:0x7f81f8b5cbe0, length:0x90 }
}
{
  real_time:646861.545873 global_time:724008, event:`SYSCALLBUF_FLUSH' tid:1221895, ticks:81750716
  { syscall:'write', ret:0xc6, size:0x10, desched:1 }
}
{
  real_time:646861.545876 global_time:724009, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:1221895, ticks:81750716
rax:0xffffffffffffffda rbx:0x7f81f835efa0 rcx:0xffffffffffffffff rdx:0xe rsi:0x3 rdi:0x4 rbp:0x7f81f0115080 rsp:0x7f81f835edf0 r8:0x0 r9:0x0 r10:0x7f81f8b5e210 r11:0x246 r12:0x7f81f0115198 r13:0x1474280 r14:0x7f81e4000b70 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x7f81f8b5f640 gs_base:0x0
}
{
  real_time:646861.545877 global_time:724010, event:`SYSCALLBUF_RESET' tid:1221895, ticks:81750716
}
{
  real_time:646861.545892 global_time:724011, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:1221895, ticks:81750716
rax:0x0 rbx:0x7f81f835efa0 rcx:0xffffffffffffffff rdx:0xe rsi:0x3 rdi:0x4 rbp:0x7f81f0115080 rsp:0x7f81f835edf0 r8:0x0 r9:0x0 r10:0x7f81f8b5e210 r11:0x246 r12:0x7f81f0115198 r13:0x1474280 r14:0x7f81e4000b70 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x7f81f8b5f640 gs_base:0x0
}
[FATAL /builddir/build/BUILD/rr-5.5.0/src/log.cc:402:emergency_debug()] (session doesn't look interactive, aborting emergency debugging)
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x5a)[0x55aae7b53e5a]
rr(_ZN2rr15notifying_abortEv+0x4f)[0x55aae7b53eef]
rr(+0x1efa58)[0x55aae7ba7a58]
rr(+0xa51b9)[0x55aae7a5d1b9]
rr(+0xa5344)[0x55aae7a5d344]
rr(_ZN2rr13RecordSession21process_syscall_entryEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultENS_13SupportedArchE+0x538)[0x55aae7a849e8]
rr(_ZN2rr13RecordSession29handle_seccomp_traced_syscallEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x3b3)[0x55aae7a7cc23]
rr(_ZN2rr13RecordSession19handle_ptrace_eventEPPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x596)[0x55aae7a81746]
rr(_ZN2rr13RecordSession11record_stepEv+0x3bf)[0x55aae7a8981f]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x87f)[0x55aae7a7a47f]
rr(main+0x12f)[0x55aae79f5edf]
/lib64/libc.so.6(+0x29590)[0x7f502a950590]
/lib64/libc.so.6(__libc_start_main+0x89)[0x7f502a950649]
rr(_start+0x25)[0x55aae79f8725]