Closed raphj closed 3 years ago
I think maybe everything is working as designed. rr intentionally propagates the segfault exit status from your tracee (bluetoothd) to itself. (This is often useful when you're using rr record
as a wrapper inside some harness script.)
No, because the daemon stops immediately in rr (the bluetooth icon does not appear in my taskbar, and rr returns immediately with a segfault and I don't have a trace), while the icon is displayed when running the daemon in rr in gdb and only crashes if I do the action that crashes (and that I am debugging).
Does service bluetooth stop; rr record /usr/libexec/bluetooth/bluetoothd
work on your system?
It doesn't segfault immediately, but the blooth icon doesn't appear, though that may be because of my configuration.
It could be a timing issue I suppose. Try running rr under strace instead of gdb?
Also, try enabling coredumps and see if you can get a coredump of rr and attach gdb to that.
You are right! It seems bluetoothd is segfaulting in rr:
[6:27:35] root@rigel /tmp # rm core
[6:27:37] root@rigel /tmp # rr record /usr/libexec/bluetooth/bluetoothd -d
rr: Saving execution to trace directory `/root/.local/share/rr/bluetoothd-14'.
[1] 28306 segmentation fault rr record /usr/libexec/bluetooth/bluetoothd -d
[6:27:43] root@rigel /tmp # gdb core
GNU gdb (GDB; openSUSE Tumbleweed) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
[New LWP 28314]
Reading symbols from /usr/libexec/bluetooth/bluetoothd...
Reading symbols from /usr/lib/debug/usr/libexec/bluetooth/bluetoothd-5.55-2.1.x86_64.debug...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/bluetooth/bluetoothd -d'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ffc922adbe5 in clock_getres ()
(gdb) l
855 "Specify an explicit path to the config file", "FILE"},
856 { "compat", 'C', 0, G_OPTION_ARG_NONE, &option_compat,
857 "Provide deprecated command line interfaces" },
858 { "experimental", 'E', 0, G_OPTION_ARG_NONE, &option_experimental,
859 "Enable experimental interfaces" },
860 { "nodetach", 'n', G_OPTION_FLAG_REVERSE,
861 G_OPTION_ARG_NONE, &option_detach,
862 "Run with logging in foreground" },
863 { "version", 'v', 0, G_OPTION_ARG_NONE, &option_version,
864 "Show version information and exit" },
(gdb) where
#0 0x00007ffc922adbe5 in clock_getres ()
#1 0x00007fa8694b1061 in __clock_getres (clock_id=clock_id@entry=1, res=res@entry=0x7ffc922500d0)
at ../sysdeps/unix/sysv/linux/clock_getres.c:37
#2 0x00007fa8696dae33 in check_monotonic_clock () at dbus-sysdeps-pthread.c:285
#3 _dbus_threads_init_platform_specific () at dbus-sysdeps-pthread.c:285
#4 dbus_threads_init (functions=<optimized out>) at dbus-threads.c:401
#5 0x00007fa8696c9465 in _dbus_lock (lock=_DBUS_LOCK_bus) at dbus-threads.c:342
#6 internal_bus_get (type=DBUS_BUS_SYSTEM, private=private@entry=0, error=error@entry=0x7ffc92250210) at dbus-bus.c:430
#7 0x00007fa8696c99ea in dbus_bus_get (type=<optimized out>, error=error@entry=0x7ffc92250210) at dbus-bus.c:561
#8 0x000055a5214e4053 in g_dbus_setup_bus (type=<optimized out>, name=0x55a52150dbd0 "org.bluez", error=0x7ffc92250210)
at gdbus/mainloop.c:291
#9 0x000055a5214642a7 in connect_dbus () at src/main.c:817
#10 main (argc=<optimized out>, argv=<optimized out>) at src/main.c:920
Here are the strace log: strace-rr-bluetoothd.log
It could be a timing issue I suppose
I would not be surprised.
In Valgrind:
[6:32:41] root@rigel /tmp # valgrind rr /usr/libexec/bluetooth/bluetoothd -d
==30622== Memcheck, a memory error detector
==30622== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30622== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==30622== Command: rr /usr/libexec/bluetooth/bluetoothd -d
==30622==
[FATAL /home/abuild/rpmbuild/BUILD/rr-5.3.0/src/PerfCounters.cc:511:check_for_xen_pmi_bug() errno: EINVAL] Overcount triggered by PMU interrupts detected due to Xen PMU virtualization bug.
Aborting. Retry with -F to override, but it will probably
fail.
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x2e)[0x295a8e]
rr(_ZN2rr15notifying_abortEv+0x4a)[0x295b0a]
rr(_ZN2rr12FatalOstreamD1Ev+0x50)[0x1bb720]
rr(+0xcdb37)[0x1d5b37]
rr(_ZN2rr12PerfCounters23default_ticks_semanticsEv+0xc)[0x1d5d6c]
rr(_ZN2rr7SessionC2Ev+0x12b)[0x26098b]
rr(_ZN2rr13RecordSessionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorIS6_SaIS6_EESD_RKNS_20DisableCPUIDFeaturesENS0_16SyscallBufferingEiNS_7BindCPUES8_PKNS_9TraceUuidE+0x31)[0x1e7061]
rr(_ZN2rr13RecordSession6createERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EESB_RKNS_20DisableCPUIDFeaturesENS0_16SyscallBufferingEhNS_7BindCPUERKS7_PKNS_9TraceUuidE+0x95d)[0x1e7d4d]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xb2e)[0x1db94e]
rr(main+0x15e)[0x15ceee]
/lib64/libc.so.6(__libc_start_main+0xea)[0x4cffe0a]
rr(_start+0x2a)[0x15d02a]
=== End rr backtrace
==30622==
==30622== Process terminating with default action of signal 6 (SIGABRT): dumping core
==30622== at 0x4D15641: raise (raise.c:50)
==30622== by 0x4CFE537: abort (abort.c:79)
==30622== by 0x295B04: rr::notifying_abort() (util.cc:1325)
==30622== by 0x1BB71F: rr::FatalOstream::~FatalOstream() (log.cc:360)
==30622== by 0x1D5B36: check_for_xen_pmi_bug (PerfCounters.cc:511)
==30622== by 0x1D5B36: check_for_bugs (PerfCounters.cc:574)
==30622== by 0x1D5B36: rr::init_attributes() (PerfCounters.cc:619)
==30622== by 0x1D5D6B: rr::PerfCounters::default_ticks_semantics() (PerfCounters.cc:654)
==30622== by 0x26098A: rr::Session::Session() (Session.cc:46)
==30622== by 0x1E7060: rr::RecordSession::RecordSession(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, rr::DisableCPUIDFeatures const&, rr::RecordSession::SyscallBuffering, int, rr::BindCPU, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rr::TraceUuid const*) (RecordSession.cc:1942)
==30622== by 0x1E7D4C: rr::RecordSession::create(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, rr::DisableCPUIDFeatures const&, rr::RecordSession::SyscallBuffering, unsigned char, rr::BindCPU, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rr::TraceUuid const*) (RecordSession.cc:1913)
==30622== by 0x1DB94D: record (RecordCommand.cc:582)
==30622== by 0x1DB94D: rr::RecordCommand::run(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (RecordCommand.cc:729)
==30622== by 0x15CEED: main (main.cc:268)
==30622==
==30622== HEAP SUMMARY:
==30622== in use at exit: 15,443 bytes in 108 blocks
==30622== total heap usage: 166 allocs, 58 frees, 131,660 bytes allocated
==30622==
==30622== LEAK SUMMARY:
==30622== definitely lost: 50 bytes in 4 blocks
==30622== indirectly lost: 0 bytes in 0 blocks
==30622== possibly lost: 0 bytes in 0 blocks
==30622== still reachable: 15,393 bytes in 104 blocks
==30622== suppressed: 0 bytes in 0 blocks
==30622== Rerun with --leak-check=full to see details of leaked memory
==30622==
==30622== For lists of detected and suppressed errors, rerun with: -s
==30622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[1] 30622 IOT instruction valgrind rr /usr/libexec/bluetooth/bluetoothd -d
So it would be a (concurrency) bug in bluetoothd triggered when running in rr not running in gdb on my machine?
Thanks for your help.
[6:36:12] root@rigel /tmp # rr replay /root/.local/share/rr/bluetoothd-14
GNU gdb (GDB; openSUSE Tumbleweed) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /root/.local/share/rr/bluetoothd-14/mmap_clone_4_bluetoothd...
Reading symbols from /usr/lib/debug/usr/libexec/bluetooth/bluetoothd-5.55-2.1.x86_64.debug...
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:31978
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/lib64/ld-2.32.so-2.32-1.1.x86_64.debug...
0x00007fa8698a1050 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) cont
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x00007ffc922adbe5 in clock_getres ()
(rr) where
#0 0x00007ffc922adbe5 in clock_getres ()
#1 0x00007fa8694b1061 in __clock_getres (clock_id=1, res=0x7ffc922500d0) at ../sysdeps/unix/sysv/linux/clock_getres.c:37
#2 0x00007fa8696dae33 in check_monotonic_clock () at dbus-sysdeps-pthread.c:285
#3 _dbus_threads_init_platform_specific () at dbus-sysdeps-pthread.c:285
#4 dbus_threads_init (functions=<optimized out>) at dbus-threads.c:401
#5 0x00007fa8696c9465 in _dbus_lock (lock=_DBUS_LOCK_bus) at dbus-threads.c:342
#6 internal_bus_get (type=DBUS_BUS_SYSTEM, private=private@entry=0, error=error@entry=0x7ffc92250210) at dbus-bus.c:430
#7 0x00007fa8696c99ea in dbus_bus_get (type=<optimized out>, error=error@entry=0x7ffc92250210) at dbus-bus.c:561
#8 0x000055a5214e4053 in g_dbus_setup_bus (type=<optimized out>, name=0x55a52150dbd0 "org.bluez", error=0x7ffc92250210)
at gdbus/mainloop.c:291
#9 0x000055a5214642a7 in connect_dbus () at src/main.c:817
#10 main (argc=<optimized out>, argv=<optimized out>) at src/main.c:920
In Valgrind:
[6:32:41] root@rigel /tmp # valgrind rr /usr/libexec/bluetooth/bluetoothd -d ==30622== Memcheck, a memory error detector ==30622== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==30622== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info ==30622== Command: rr /usr/libexec/bluetooth/bluetoothd -d ==30622== [FATAL /home/abuild/rpmbuild/BUILD/rr-5.3.0/src/PerfCounters.cc:511:check_for_xen_pmi_bug() errno: EINVAL] Overcount triggered by PMU interrupts detected due to Xen PMU virtualization bug. Aborting. Retry with -F to override, but it will probably fail.
This is just valgrind breaking rr, it's not real.
Which version of rr is this?
Can you run disassemble
in the rr replay and figure out what is causing the segfault?
Do the rr tests pass on your system?
[8:05:07] root@rigel /tmp # rr replay /root/.local/share/rr/bluetoothd-14 [69]
GNU gdb (GDB; openSUSE Tumbleweed) 9.2
[...]
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /root/.local/share/rr/bluetoothd-14/mmap_clone_4_bluetoothd...
Reading symbols from /usr/lib/debug/usr/libexec/bluetooth/bluetoothd-5.55-2.1.x86_64.debug...
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:4330
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/lib64/ld-2.32.so-2.32-1.1.x86_64.debug...
0x00007fa8698a1050 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) disassemble
Dump of assembler code for function _start:
=> 0x00007fa8698a1050 <+0>: mov %rsp,%rdi
0x00007fa8698a1053 <+3>: callq 0x7fa8698a1d80 <_dl_start>
End of assembler dump.
# rr --version
rr version 5.3.0
I'm going to find out how to run the tests and come back later with the results.
Sorry, my last comment was not helpful:
0x00007fa8698a1050 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) cont
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x00007ffc922adbe5 in clock_getres ()
(rr) disassemble
Dump of assembler code for function clock_getres:
0x00007ffc922adbd0 <+0>: cmp $0xf,%edi
0x00007ffc922adbd3 <+3>: ja 0x7ffc922adc25 <clock_getres+85>
0x00007ffc922adbd5 <+5>: lea -0x4b5c(%rip),%rax # 0x7ffc922a9080
0x00007ffc922adbdc <+12>: lea -0x1b63(%rip),%rdx # 0x7ffc922ac080
0x00007ffc922adbe3 <+19>: mov %edi,%ecx
=> 0x00007ffc922adbe5 <+21>: cmpl $0x7fffffff,-0x4b6b(%rip) # 0x7ffc922a9084
0x00007ffc922adbef <+31>: cmovne %rax,%rdx
0x00007ffc922adbf3 <+35>: mov $0x1,%eax
0x00007ffc922adbf8 <+40>: shl %cl,%eax
0x00007ffc922adbfa <+42>: test $0x893,%eax
0x00007ffc922adbff <+47>: jne 0x7ffc922adc1d <clock_getres+77>
0x00007ffc922adc01 <+49>: test $0x60,%al
0x00007ffc922adc03 <+51>: je 0x7ffc922adc25 <clock_getres+85>
0x00007ffc922adc05 <+53>: mov $0x3d0900,%eax
0x00007ffc922adc0a <+58>: test %rsi,%rsi
0x00007ffc922adc0d <+61>: je 0x7ffc922adc2d <clock_getres+93>
0x00007ffc922adc0f <+63>: mov %rax,0x8(%rsi)
0x00007ffc922adc13 <+67>: xor %eax,%eax
0x00007ffc922adc15 <+69>: movq $0x0,(%rsi)
0x00007ffc922adc1c <+76>: retq
0x00007ffc922adc1d <+77>: mov 0xe8(%rdx),%eax
0x00007ffc922adc23 <+83>: jmp 0x7ffc922adc0a <clock_getres+58>
0x00007ffc922adc25 <+85>: mov $0xe5,%eax
0x00007ffc922adc2a <+90>: syscall
0x00007ffc922adc2c <+92>: retq
0x00007ffc922adc2d <+93>: xor %eax,%eax
0x00007ffc922adc2f <+95>: retq
End of assembler dump.
These tests fail on my machine on master (commit 99cc4f29acea5c794a002e7f7582b5b6d9e32e36):
824 - x86/cpuid_singlestep (Failed)
825 - x86/cpuid_singlestep-no-syscallbuf (Failed)
1264 - adjtimex-32 (Failed)
1265 - adjtimex-32-no-syscallbuf (Failed)
1284 - barrier-32 (Failed)
1285 - barrier-32-no-syscallbuf (Failed)
1286 - big_buffers-32 (Failed)
1287 - big_buffers-32-no-syscallbuf (Failed)
1288 - block-32 (Failed)
1289 - block-32-no-syscallbuf (Failed)
1306 - clock-32 (Failed)
1307 - clock-32-no-syscallbuf (Failed)
1308 - clock_adjtime-32 (Failed)
1309 - clock_adjtime-32-no-syscallbuf (Failed)
1338 - constructor-32 (Failed)
1339 - constructor-32-no-syscallbuf (Failed)
1444 - gettimeofday-32 (Failed)
1445 - gettimeofday-32-no-syscallbuf (Failed)
1460 - intr_futex_wait_restart-32 (Failed)
1461 - intr_futex_wait_restart-32-no-syscallbuf (Failed)
1462 - intr_poll-32 (Failed)
1463 - intr_poll-32-no-syscallbuf (Failed)
1468 - intr_read_no_restart-32 (Failed)
1469 - intr_read_no_restart-32-no-syscallbuf (Failed)
1470 - intr_read_restart-32 (Failed)
1471 - intr_read_restart-32-no-syscallbuf (Failed)
1474 - intr_sleep_no_restart-32 (Failed)
1475 - intr_sleep_no_restart-32-no-syscallbuf (Failed)
1542 - mmap_shared-32 (Failed)
1543 - mmap_shared-32-no-syscallbuf (Failed)
1546 - mmap_shared_grow-32 (Failed)
1547 - mmap_shared_grow-32-no-syscallbuf (Failed)
1552 - mmap_shared_subpage-32 (Failed)
1553 - mmap_shared_subpage-32-no-syscallbuf (Failed)
1578 - mq-32 (Failed)
1579 - mq-32-no-syscallbuf (Failed)
1740 - readlink-32 (Failed)
1741 - readlink-32-no-syscallbuf (Failed)
1792 - seccomp_sigsys_syscallbuf-32 (Failed)
1793 - seccomp_sigsys_syscallbuf-32-no-syscallbuf (Failed)
1922 - switch_read-32 (Failed)
1923 - switch_read-32-no-syscallbuf (Failed)
1984 - unshare-32 (Failed)
1985 - unshare-32-no-syscallbuf (Failed)
1986 - utimes-32 (Failed)
1987 - utimes-32-no-syscallbuf (Failed)
2024 - async_signal_syscalls-32 (Failed)
2026 - async_signal_syscalls2-32 (Failed)
2028 - async_signal_syscalls_siginfo-32 (Failed)
2054 - call_function-32 (Failed)
2055 - call_function-32-no-syscallbuf (Failed)
2058 - checkpoint_mixed_mode-32 (Failed)
2059 - checkpoint_mixed_mode-32-no-syscallbuf (Failed)
2074 - x86/cpuid_singlestep-32 (Failed)
2075 - x86/cpuid_singlestep-32-no-syscallbuf (Failed)
2104 - exit_race-32 (Failed)
2105 - exit_race-32-no-syscallbuf (Failed)
2112 - fork_syscalls-32 (Failed)
2113 - fork_syscalls-32-no-syscallbuf (Failed)
2140 - intr_ptrace_decline-32 (Failed)
2141 - intr_ptrace_decline-32-no-syscallbuf (Failed)
2254 - sysconf_onln-32 (Failed)
2255 - sysconf_onln-32-no-syscallbuf (Failed)
2272 - thread_stress-32 (Failed)
2273 - thread_stress-32-no-syscallbuf (Failed)
2276 - threads-32 (Failed)
2277 - threads-32-no-syscallbuf (Failed)
2286 - vdso_gettimeofday_stack-32 (Failed)
2288 - vdso_clock_gettime_stack-32 (Failed)
2314 - async_signal_syscalls_100-32 (Failed)
2315 - async_signal_syscalls_100-32-no-syscallbuf (Failed)
2316 - async_signal_syscalls_1000-32 (Failed)
2317 - async_signal_syscalls_1000-32-no-syscallbuf (Failed)
2320 - break_block-32 (Failed)
2321 - break_block-32-no-syscallbuf (Failed)
2322 - break_clock-32 (Failed)
2323 - break_clock-32-no-syscallbuf (Failed)
2340 - break_thread-32 (Failed)
2341 - break_thread-32-no-syscallbuf (Failed)
2364 - dead_thread_target-32 (Failed)
2365 - dead_thread_target-32-no-syscallbuf (Failed)
2368 - deliver_async_signal_during_syscalls-32 (Failed)
2386 - fork_exec_info_thr-32 (Failed)
2387 - fork_exec_info_thr-32-no-syscallbuf (Failed)
2388 - get_thread_list-32 (Failed)
2389 - get_thread_list-32-no-syscallbuf (Failed)
2390 - hardlink_mmapped_files-32 (Failed)
2391 - hardlink_mmapped_files-32-no-syscallbuf (Failed)
All with the following error:
***Failed Error regular expression found in output. Regex=[FAILED]
Run info proc map
. Is 0x00007ffc922adbe5 in the [vdso] mapping?
Run RR_LOG=Monkeypatcher rr record ...
and dump the results here?
I just tried to use rr compiled from master
(to run the tests yesterday), and things seem to work fine (no immediate crash), so this might be a bug from the openSUSE package!
anyway, here are the results of the commands you asked with the openSUSE rr:
It seems 0x00007ffc922adbe5 is in the [vdso] mapping:
0x00007fa8698a1050 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) cont
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x00007ffc922adbe5 in clock_getres ()
(rr) info proc map
process 28314
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0x68000000 0x68200000 0x200000 0x0
0x70000000 0x70001000 0x1000 0x0 /usr/bin/rr_page_64_replay
0x70001000 0x70002000 0x1000 0x0 /tmp/rr-shared-preload_thread_locals-6547-0
0x55a521444000 0x55a521462000 0x1e000 0x0 /usr/libexec/bluetooth/bluetoothd
0x55a521462000 0x55a521503000 0xa1000 0x1e000 /usr/libexec/bluetooth/bluetoothd
0x55a521503000 0x55a521545000 0x42000 0xbf000 /usr/libexec/bluetooth/bluetoothd
0x55a521545000 0x55a52154e000 0x9000 0x100000 /usr/libexec/bluetooth/bluetoothd
0x55a52154e000 0x55a521553000 0x5000 0x109000 /usr/libexec/bluetooth/bluetoothd
0x55a5215b6000 0x55a5215d7000 0x21000 0x0 [heap]
0x7fa868e8b000 0x7fa868f8b000 0x100000 0x0 /tmp/rr-shared-syscallbuf.28314-6547-3
0x7fa868f8b000 0x7fa868f90000 0x5000 0x0
0x7fa868f90000 0x7fa868f95000 0x5000 0x0 /usr/lib64/libgpg-error.so.0.30.0
0x7fa868f95000 0x7fa868faa000 0x15000 0x5000 /usr/lib64/libgpg-error.so.0.30.0
0x7fa868faa000 0x7fa868fb3000 0x9000 0x1a000 /usr/lib64/libgpg-error.so.0.30.0
0x7fa868fb3000 0x7fa868fb4000 0x1000 0x22000 /usr/lib64/libgpg-error.so.0.30.0
0x7fa868fb4000 0x7fa868fb5000 0x1000 0x23000 /usr/lib64/libgpg-error.so.0.30.0
0x7fa868fb5000 0x7fa868fc2000 0xd000 0x0 /usr/lib64/libgcrypt.so.20.2.6
0x7fa868fc2000 0x7fa86909a000 0xd8000 0xd000 /usr/lib64/libgcrypt.so.20.2.6
0x7fa86909a000 0x7fa8690d3000 0x39000 0xe5000 /usr/lib64/libgcrypt.so.20.2.6
0x7fa8690d3000 0x7fa8690d5000 0x2000 0x11d000 /usr/lib64/libgcrypt.so.20.2.6
0x7fa8690d5000 0x7fa8690db000 0x6000 0x11f000 /usr/lib64/libgcrypt.so.20.2.6
0x7fa8690db000 0x7fa8690dc000 0x1000 0x0
0x7fa8690dc000 0x7fa8690df000 0x3000 0x0 /usr/lib64/liblz4.so.1.9.2
0x7fa8690df000 0x7fa8690f8000 0x19000 0x3000 /usr/lib64/liblz4.so.1.9.2
0x7fa8690f8000 0x7fa8690fb000 0x3000 0x1c000 /usr/lib64/liblz4.so.1.9.2
0x7fa8690fb000 0x7fa8690fc000 0x1000 0x1e000 /usr/lib64/liblz4.so.1.9.2
0x7fa8690fc000 0x7fa8690fd000 0x1000 0x1f000 /usr/lib64/liblz4.so.1.9.2
--Type <RET> for more, q to quit, c to continue without paging--
0x7fa8690fd000 0x7fa8690ff000 0x2000 0x0
0x7fa8690ff000 0x7fa869102000 0x3000 0x0 /usr/lib64/liblzma.so.5.2.5
0x7fa869102000 0x7fa869124000 0x22000 0x3000 /usr/lib64/liblzma.so.5.2.5
0x7fa869124000 0x7fa86912f000 0xb000 0x25000 /usr/lib64/liblzma.so.5.2.5
0x7fa86912f000 0x7fa869130000 0x1000 0x30000 /usr/lib64/liblzma.so.5.2.5
0x7fa869130000 0x7fa869131000 0x1000 0x30000 /usr/lib64/liblzma.so.5.2.5
0x7fa869131000 0x7fa869132000 0x1000 0x31000 /usr/lib64/liblzma.so.5.2.5
0x7fa869132000 0x7fa869135000 0x3000 0x0 /lib64/librt-2.32.so
0x7fa869135000 0x7fa869139000 0x4000 0x3000 /lib64/librt-2.32.so
0x7fa869139000 0x7fa86913b000 0x2000 0x7000 /lib64/librt-2.32.so
0x7fa86913b000 0x7fa86913c000 0x1000 0x8000 /lib64/librt-2.32.so
0x7fa86913c000 0x7fa86913d000 0x1000 0x9000 /lib64/librt-2.32.so
0x7fa86913d000 0x7fa86914c000 0xf000 0x0 /lib64/libm-2.32.so
0x7fa86914c000 0x7fa8691e7000 0x9b000 0xf000 /lib64/libm-2.32.so
0x7fa8691e7000 0x7fa869280000 0x99000 0xaa000 /lib64/libm-2.32.so
0x7fa869280000 0x7fa869281000 0x1000 0x142000 /lib64/libm-2.32.so
0x7fa869281000 0x7fa869282000 0x1000 0x143000 /lib64/libm-2.32.so
0x7fa869282000 0x7fa869295000 0x13000 0x0 /usr/lib64/libsystemd.so.0.29.0
0x7fa869295000 0x7fa86930d000 0x78000 0x13000 /usr/lib64/libsystemd.so.0.29.0
0x7fa86930d000 0x7fa869332000 0x25000 0x8b000 /usr/lib64/libsystemd.so.0.29.0
0x7fa869332000 0x7fa869336000 0x4000 0xaf000 /usr/lib64/libsystemd.so.0.29.0
0x7fa869336000 0x7fa869337000 0x1000 0xb3000 /usr/lib64/libsystemd.so.0.29.0
0x7fa869337000 0x7fa869338000 0x1000 0x0
0x7fa869338000 0x7fa86933a000 0x2000 0x0 /usr/lib64/libpcre.so.1.2.12
0x7fa86933a000 0x7fa8693a6000 0x6c000 0x2000 /usr/lib64/libpcre.so.1.2.12
0x7fa8693a6000 0x7fa8693c5000 0x1f000 0x6e000 /usr/lib64/libpcre.so.1.2.12
0x7fa8693c5000 0x7fa8693c6000 0x1000 0x8c000 /usr/lib64/libpcre.so.1.2.12
0x7fa8693c6000 0x7fa8693c7000 0x1000 0x8d000 /usr/lib64/libpcre.so.1.2.12
0x7fa8693c7000 0x7fa8693ce000 0x7000 0x0 /lib64/libpthread-2.32.so
0x7fa8693ce000 0x7fa8693dd000 0xf000 0x7000 /lib64/libpthread-2.32.so
0x7fa8693dd000 0x7fa8693e2000 0x5000 0x16000 /lib64/libpthread-2.32.so
--Type <RET> for more, q to quit, c to continue without paging--
0x7fa8693e2000 0x7fa8693e3000 0x1000 0x1a000 /lib64/libpthread-2.32.so
0x7fa8693e3000 0x7fa8693e4000 0x1000 0x1b000 /lib64/libpthread-2.32.so
0x7fa8693e4000 0x7fa8693ea000 0x6000 0x0
0x7fa8693ea000 0x7fa869410000 0x26000 0x0 /lib64/libc-2.32.so
0x7fa869410000 0x7fa86955c000 0x14c000 0x26000 /lib64/libc-2.32.so
0x7fa86955c000 0x7fa8695a8000 0x4c000 0x172000 /lib64/libc-2.32.so
0x7fa8695a8000 0x7fa8695a9000 0x1000 0x1be000 /lib64/libc-2.32.so
0x7fa8695a9000 0x7fa8695ac000 0x3000 0x1be000 /lib64/libc-2.32.so
0x7fa8695ac000 0x7fa8695af000 0x3000 0x1c1000 /lib64/libc-2.32.so
0x7fa8695af000 0x7fa8695b3000 0x4000 0x0
0x7fa8695b3000 0x7fa8695df000 0x2c000 0x0 /usr/lib64/libasound.so.2.0.0
0x7fa8695df000 0x7fa86966a000 0x8b000 0x2c000 /usr/lib64/libasound.so.2.0.0
0x7fa86966a000 0x7fa8696a4000 0x3a000 0xb7000 /usr/lib64/libasound.so.2.0.0
0x7fa8696a4000 0x7fa8696aa000 0x6000 0xf0000 /usr/lib64/libasound.so.2.0.0
0x7fa8696aa000 0x7fa8696ab000 0x1000 0xf6000 /usr/lib64/libasound.so.2.0.0
0x7fa8696ab000 0x7fa8696ac000 0x1000 0x0 /lib64/libdl-2.32.so
0x7fa8696ac000 0x7fa8696ae000 0x2000 0x1000 /lib64/libdl-2.32.so
0x7fa8696ae000 0x7fa8696af000 0x1000 0x3000 /lib64/libdl-2.32.so
0x7fa8696af000 0x7fa8696b0000 0x1000 0x3000 /lib64/libdl-2.32.so
0x7fa8696b0000 0x7fa8696b1000 0x1000 0x4000 /lib64/libdl-2.32.so
0x7fa8696b1000 0x7fa8696bf000 0xe000 0x0 /usr/lib64/libdbus-1.so.3.19.13
0x7fa8696bf000 0x7fa8696ed000 0x2e000 0xe000 /usr/lib64/libdbus-1.so.3.19.13
0x7fa8696ed000 0x7fa8696ff000 0x12000 0x3c000 /usr/lib64/libdbus-1.so.3.19.13
0x7fa8696ff000 0x7fa869700000 0x1000 0x4e000 /usr/lib64/libdbus-1.so.3.19.13
0x7fa869700000 0x7fa869701000 0x1000 0x4e000 /usr/lib64/libdbus-1.so.3.19.13
0x7fa869701000 0x7fa869702000 0x1000 0x4f000 /usr/lib64/libdbus-1.so.3.19.13
0x7fa869702000 0x7fa869703000 0x1000 0x0
0x7fa869703000 0x7fa869722000 0x1f000 0x0 /usr/lib64/libglib-2.0.so.0.6400.6
0x7fa869722000 0x7fa8697a6000 0x84000 0x1f000 /usr/lib64/libglib-2.0.so.0.6400.6
0x7fa8697a6000 0x7fa86982b000 0x85000 0xa3000 /usr/lib64/libglib-2.0.so.0.6400.6
0x7fa86982b000 0x7fa86982c000 0x1000 0x127000 /usr/lib64/libglib-2.0.so.0.6400.6
--Type <RET> for more, q to quit, c to continue without paging--
0x7fa86982c000 0x7fa86982d000 0x1000 0x128000 /usr/lib64/libglib-2.0.so.0.6400.6
0x7fa86982d000 0x7fa86982e000 0x1000 0x0
0x7fa869865000 0x7fa86986c000 0x7000 0x0 /usr/lib64/gconv/gconv-modules.cache
0x7fa86986c000 0x7fa86986d000 0x1000 0x0 /usr/lib64/rr/librrpreload.so
0x7fa86986d000 0x7fa869893000 0x26000 0x1000 /usr/lib64/rr/librrpreload.so
0x7fa869893000 0x7fa869896000 0x3000 0x27000 /usr/lib64/rr/librrpreload.so
0x7fa869896000 0x7fa869897000 0x1000 0x29000 /usr/lib64/rr/librrpreload.so
0x7fa869897000 0x7fa869898000 0x1000 0x2a000 /usr/lib64/rr/librrpreload.so
0x7fa869898000 0x7fa8698a0000 0x8000 0x0
0x7fa8698a0000 0x7fa8698a1000 0x1000 0x0 /lib64/ld-2.32.so
0x7fa8698a1000 0x7fa8698c2000 0x21000 0x1000 /lib64/ld-2.32.so
0x7fa8698c2000 0x7fa8698ca000 0x8000 0x22000 /lib64/ld-2.32.so
0x7fa8698cb000 0x7fa8698cc000 0x1000 0x2a000 /lib64/ld-2.32.so
0x7fa8698cc000 0x7fa8698ce000 0x2000 0x2b000 /lib64/ld-2.32.so
0x7fa8698cf000 0x7fa8698d0000 0x1000 0x0
0x7ffc92230000 0x7ffc92252000 0x22000 0x0 [stack]
0x7ffc922ad000 0x7ffc922af000 0x2000 0x0 [vdso]
0x7ffc922b0000 0x7ffc922b1000 0x1000 0x0
0xffffffffff600000 0xffffffffff601000 0x1000 0x0 [vsyscall]
# RR_LOG=Monkeypatcher rr record /usr/libexec/bluetooth/bluetoothd -d rr: Saving execution to trace directory `/root/.local/share/rr/bluetoothd-16'.
[Monkeypatcher] monkeypatched __vdso_clock_gettime to syscall 228 at 0x7fff6e1cdbc0 (0xbc0)
[Monkeypatcher] monkeypatched __vdso_gettimeofday to syscall 96 at 0x7fff6e1cdb80 (0xb80)
[Monkeypatcher] monkeypatched __vdso_time to syscall 201 at 0x7fff6e1cdb90 (0xb90)
[Monkeypatcher] monkeypatched __vdso_getcpu to syscall 309 at 0x7fff6e1cdc30 (0xc30)
[Monkeypatcher] Patched syscall at 0x7fe3149902d7 syscall umask tid 7124 bytes 0xc3 0xf 0x1f 0x84 0 0 0 0 0 0xb8 0x5a 0 0 0
[Monkeypatcher] Patched syscall at 0x7fe31499a799 syscall futex tid 7124 bytes 0x48 0x3d 0x1 0xf0 0xff 0xff 0x73 0x1 0xc3 0x48 0x8b 0xd 0xa7 0x76
[Monkeypatcher] Patched syscall at 0x7fe31499fec7 syscall eventfd2 tid 7124 bytes 0x48 0x3d 0x1 0xf0 0xff 0xff 0x73 0x1 0xc3 0x48 0x8b 0xd 0x79 0x1f
[Monkeypatcher] Patched syscall at 0x7fe314990943 syscall write tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x55 0xc3 0xf 0x1f 0x40 0 0x48
[Monkeypatcher] Patched syscall at 0x7fe3149a1357 syscall socket tid 7124 bytes 0x48 0x3d 0x1 0xf0 0xff 0xff 0x73 0x1 0xc3 0x48 0x8b 0xd 0xe9 0xa
[Monkeypatcher] Patched syscall at 0x7fe3149a0d27 syscall bind tid 7124 bytes 0x48 0x3d 0x1 0xf0 0xff 0xff 0x73 0x1 0xc3 0x48 0x8b 0xd 0x19 0x11
[Monkeypatcher] Patched syscall at 0x7fe3149a0d63 syscall connect tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x55 0xc3 0xf 0x1f 0x40 0 0x48
[Monkeypatcher] Patched syscall at 0x7fff6e1cdbc7 syscall clock_gettime tid 7124 bytes 0x90 0x90 0x90 0xc3 0 0 0 0 0 0x83 0xff 0xf 0x77 0x50
[Monkeypatcher] Patched syscall at 0x7fe314995c18 syscall openat tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x40 0x48 0x8b 0x4c 0x24 0x18 0x64
[Monkeypatcher] Patched syscall at 0x7fe31498fcc5 syscall fstat tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x3 0xc3 0x66 0x90 0x48 0x8b 0x15
[Monkeypatcher] Patched syscall at 0x7fe314995d24 syscall read tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x4 0xc3 0xf 0x1f 0 0x48 0x8b
[Monkeypatcher] Patched syscall at 0x7fe3149909d7 syscall lseek tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x1 0xc3 0x48 0x8b 0x15 0x69 0x14
[Monkeypatcher] Patched syscall at 0x7fe314995a77 syscall close tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x1 0xc3 0x48 0x8b 0x15 0xc9 0xc3
[Monkeypatcher] Patched syscall at 0x7fe31496d3a7 syscall getpid tid 7124 bytes 0xc3 0xf 0x1f 0x84 0 0 0 0 0 0xb8 0x6e 0 0 0
[Monkeypatcher] Patched syscall at 0x7fe3149a10ec syscall sendto tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x64 0xc3 0xf 0x1f 0 0x55 0x48
[Monkeypatcher] Patched syscall at 0x7fe3149a11a3 syscall sendmsg tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x55 0xc3 0xf 0x1f 0x40 0 0x48
[Monkeypatcher] Patched syscall at 0x7fe3149905f7 syscall openat tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0xf 0x87 0x95 0 0 0 0x48 0x8b
[Monkeypatcher] Patched syscall at 0x7fe31499089e syscall read tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x5a 0xc3 0x66 0xf 0x1f 0x84 0
[Monkeypatcher] Patched syscall at 0x7fe31499a942 syscall mmap tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0x77 0x66 0x5b 0x5d 0xc3 0xf 0x1f 0
[Monkeypatcher] Patched syscall at 0x7fe31488dae7 syscall futex tid 7124 bytes 0x48 0x3d 0 0xf0 0xff 0xff 0xf 0x86 0x3e 0xff 0xff 0xff 0x83 0xc0
[1] 7116 segmentation fault RR_LOG=Monkeypatcher rr record /usr/libexec/bluetooth/bluetoothd -d
I should have had the presence of mind to check this bug with rr as compiled from the repository, sorry for this!
Actually I think this was fixed by 4287a0c3b1c096c60621fa3bd220853ce0ba5cb0 which I just landed about two hours ago so building from master before then wouldn't have helped you :-).
mhm, probably not since I built master yesterday :-)
Nice anyway, and hopefully this issue will be helpful as a documentation. Have a good day!
I am studying a bug (segfault) in
/usr/libexec/bluetooth/bluetoothd
on openSUSE Tumbleweed (fully updated).The following command ran as root segfaults (full logs):
But when run in gdb to debug the rr segfault, rr succeeds in getting a record of my faulty bluetoothd execution:
So I don't really know how I can help.
I'm using rr as provided by openSUSE Tumbleweed, on an Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz running Linux 5.8.10-1-default (x86_64).
Let me know if you need me to try something.