Segmentation Fault running example/trace-syscalls-ld.so

[evandrix]

was just trying to follow along the article https://www.humprog.org/~stephen/blog/2021/10/14/#syscall-tracing-in-process, and reproducing what was written on my host:

• Ubuntu 20.04.3
• Linux <hostname> 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• cc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

Managed to build, as recommended in the README.md,

• contrib/Makefile
• src/Makefile
• lib/Makefile (btw, this was missing in the README?)
• example/Makefile

---

$ LD_PRELOAD=$(pwd)/trace-syscalls.so /bin/true
TRACE_SYSCALLS_TRACE_FD is (none), dup'ing stderr, fd 3 is open; outputting traces there.
xed could not decode instruction at 0x00007fff35b799c7
xed could not decode instruction at 0x00007fff35b799d4
xed could not decode instruction at 0x00007fff35b799e1
xed could not decode instruction at 0x00007fff35b799ee
xed could not decode instruction at 0x00007fff35b799ef
xed could not decode instruction at 0x00007fff35b799fb
xed could not decode instruction at 0x00007fff35b79a08
xed could not decode instruction at 0x00007fff35b79a15
== 293166 == > 0x7f4bdd78f2c4 (/lib/x86_64-linux-gnu/libc.so.6+0xe62c4) exit_group(0, 0x3c, 0, 0x8, 0xe7, 0xffffffffffffff80)

i guess this is fine, and those lines before the last line are just warnings in the output that can be ignored/discarded?

edit: this works

$ TRACE_SYSCALLS_TRACE_FD=1 LD_PRELOAD=$(pwd)/trace-syscalls.so /bin/true 2>/dev/null
== 295960 == > 0x7fec064b22c4 (/lib/x86_64-linux-gnu/libc.so.6+0xe62c4) exit_group(0, 0x3c, 0, 0x8, 0xe7, 0xffffffffffffff80)

---

$ ./trace-syscalls-ld.so /bin/true
Hello from trace-syscalls-ld!
We think we are the program
AT_PHDR is 0x555555556040
AT_PHENT is 0x38
AT_PHNUM is 0xb
AT_BASE is 0
AT_ENTRY is 0x555555557100
AT_EXECFN is 0x7ffceb6913dd (./trace-syscalls-ld.so)
[1]    293258 segmentation fault (core dumped)  ./trace-syscalls-ld.so /bin/true

not sure what i'm missing here, was expecting to see sycall output, similar to strace

[stephenrkell]

Thanks for reporting this. You're right that the output showing exit_group is correct (although it would be nice to fix the xed decode problem, that is separate).

What I am getting from your report is that the LD_PRELOAD case works but the -ld.so case does not work for you. If you could get me a backtrace on the latter, that would be helpful. Just running under gdb and doing 'bt' may be enough, although this gets tricky with custom loaders... depending on where the crash is, you may need to use 'add-symbol-file' to tell it about the actual program (/bin/true).

[stephenrkell]

... or (silly me for forgetting) the actual dynamic linker: /lib64/ld-linux-x86-64.so.2. Currently this is always loaded at 0x555555556000, which is not great security-wise but is handy for debugging.

[evandrix]

Thanks for reporting this. You're right that the output showing exit_group is correct (although it would be nice to fix the xed decode problem, that is separate).

What I am getting from your report is that the LD_PRELOAD case works but the -ld.so case does not work for you. If you could get me a backtrace on the latter, that would be helpful. Just running under gdb and doing 'bt' may be enough, although this gets tricky with custom loaders... depending on where the crash is, you may need to use 'add-symbol-file' to tell it about the actual program (/bin/true).

$ gdb ./trace-syscalls-ld.so
GNU gdb (Ubuntu 10.2-0ubuntu1~20.04~1) 10.2
(gdb) add-symbol-file /bin/true
(gdb) r /bin/true
Starting program: /libsystrap/example/trace-syscalls-ld.so /bin/true
Hello from trace-syscalls-ld!
We think we are the program
AT_PHDR is 0x555555556040
AT_PHENT is 0x38
AT_PHNUM is 0xb
AT_BASE is 0
AT_ENTRY is 0x555555557100
AT_EXECFN is 0x7fffffffe380 (/libsystrap/example/trace-syscalls-ld.so)

Program received signal SIGSEGV, Segmentation fault.
frob_dynamic (inferior_load_addr=inferior_load_addr@entry=93824992239616, inferior_dynamic_vaddr=inferior_dynamic_vaddr@entry=188024,
    phdrs=phdrs@entry=0x7fffffffdc18, phnum=<optimized out>) at chain.c:181
181                 r->r_version = 0;
(gdb) bt
#0  frob_dynamic (inferior_load_addr=inferior_load_addr@entry=93824992239616, inferior_dynamic_vaddr=inferior_dynamic_vaddr@entry=188024,
    phdrs=phdrs@entry=0x7fffffffdc18, phnum=<optimized out>) at chain.c:181
#1  0x00007ffff7bc599b in main (argc=<optimized out>, argv=<optimized out>) at /libsystrap/contrib/donald/src/main.c:232

https://github.com/stephenrkell/libsystrap/blob/master/example/chain.c#L181

[stephenrkell]

Thanks! This is some especially nasty code that is ironically there to make debugging work.

Clearly the calculation

inferior_load_addr + found_ldso_r_debug->st_value

has not yielded a valid address, when it is supposed to give you the address of the _r_debug symbol in the "inferior" i.e. the normal dynamic linker. So this part is actually pretty routine, just looking up the _r_debug symbol... hmm.

If you could print found_ldso_r_debug and *found_ldso_r_debug it may be useful, and also post the contents of the proc maps file (use info proc stat to get the pid, then shell cat /proc/$pid/maps). You could also e-mail me the dynamic linker binary from your system.

[evandrix]

Thanks! This is some especially nasty code that is ironically there to make debugging work.

Clearly the calculation

inferior_load_addr + found_ldso_r_debug->st_value

has not yielded a valid address, when it is supposed to give you the address of the _r_debug symbol in the "inferior" i.e. the normal dynamic linker. So this part is actually pretty routine, just looking up the _r_debug symbol... hmm.

If you could print found_ldso_r_debug and *found_ldso_r_debug it may be useful, and also post the contents of the proc maps file (use info proc stat to get the pid, then shell cat /proc/$pid/maps). You could also e-mail me the dynamic linker binary from your system.

(gdb) p found_ldso_r_debug
$1 = (Elf64_Sym *) 0x5555555565e0

(gdb) p *found_ldso_r_debug
$2 = {st_name = 456, st_info = 17 '\021', st_other = 0 '\000', st_shndx = 24, st_value = 192864, st_size = 40}

(gdb) info proc stat
process 642081
Process: 642081
Exec file: trace-syscalls-ld.so
State: t
Parent process: 642048
Process group: 642081
Session id: 641857
TTY: 34818
TTY owner process group: 642048
Flags: 0x40000000
Minor faults (no memory page): 345
Minor faults, children: 0
Major faults (memory page faults): 0
Major faults, children: 0
utime: 0
stime: 0
utime, children: 0
stime, children: 0
jiffies remaining in current time slice: 20
'nice' value: 0
jiffies until next timeout: 1
jiffies until next SIGALRM: 0
start time (jiffies since system boot): 107398601
Virtual memory size: 5758976
Resident set size: 441
rlim: 18446744073709551615
Start of text: 0x7ffff7ad6000
End of text: 0x7ffff7e3b0c0
Start of stack: 0x7fffffffe070

(gdb) shell cat /proc/642081/maps
555555556000-555555557000 r--p 00000000 fd:00 133865     /usr/lib/x86_64-linux-gnu/ld-2.31.so
555555557000-55555557a000 r-xp 00001000 fd:00 133865     /usr/lib/x86_64-linux-gnu/ld-2.31.so
55555557a000-555555582000 r--p 00024000 fd:00 133865     /usr/lib/x86_64-linux-gnu/ld-2.31.so
555555582000-555555583000 ---p 0002c000 fd:00 133865     /usr/lib/x86_64-linux-gnu/ld-2.31.so
555555583000-555555585000 rw-p 0002c000 fd:00 133865     /usr/lib/x86_64-linux-gnu/ld-2.31.so
555555585000-555555586000 ---p 0002f000 fd:00 133865     /usr/lib/x86_64-linux-gnu/ld-2.31.so
7ffff7ad2000-7ffff7ad5000 r--p 00000000 00:00 0          [vvar]
7ffff7ad5000-7ffff7ad6000 r-xp 00000000 00:00 0          [vdso]
7ffff7ad6000-7ffff7bc5000 r--p 00000000 fd:00 5001305    /libsystrap/example/trace-syscalls-ld.so
7ffff7bc5000-7ffff7e3c000 r-xp 000ef000 fd:00 5001305    /libsystrap/example/trace-syscalls-ld.so
7ffff7e3c000-7ffff7f56000 r--p 00366000 fd:00 5001305    /libsystrap/example/trace-syscalls-ld.so
7ffff7f56000-7ffff7fdc000 rw-p 0047f000 fd:00 5001305    /libsystrap/example/trace-syscalls-ld.so
7ffff7fdc000-7ffff7fff000 rw-p 00000000 00:00 0          [heap]
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0  [vsyscall]

ld-2.31.so.zip

[stephenrkell]

Hmm. That's interesting... there's no permission at all on the last page of the data segment. It may be a bug in my segment-mapping code. I'll take a closer look tomorrow.

[stephenrkell]

I've just pushed what I hope is a fix. Make sure your pull updates the 'donald' submodule. And you should probably 'make clean' in example/, as I can't remember if the make deps are complete enough to see updates in donald. Let me know how you get on.

[evandrix]

okay yes, turns out i probably didn't clean it properly ... i deleted the whole directory, and re-did everything from scratch, and it works!!

$ TRACE_SYSCALLS_TRACE_FD=1 LD_PRELOAD=$(pwd)/trace-syscalls.so /bin/true 2>/dev/null
== 915678 == > 0x7f7b233212c4 (/lib/x86_64-linux-gnu/libc.so.6+0xe62c4) exit_group(0, 0x3c, 0, 0x8, 0xe7, 0xffffffffffffff80)

$ TRACE_SYSCALLS_TRACE_FD=1 trace-syscalls-ld.so /bin/true
Hello from trace-syscalls-ld!
We think we are the program
AT_PHDR is 0x555555556040
AT_PHENT is 0x38
AT_PHNUM is 0xb
AT_BASE is 0
AT_ENTRY is 0x555555557100
AT_EXECFN is 0x7ffdf73203c5 (trace-syscalls-ld.so)
Decoding 837 bytes from vaddr 0x680
We saw an instruction at vdso vaddr 0x731 with len 7 displacement -5944 memoplen 8
We saw an instruction at vdso vaddr 0x744 with len 7 displacement -5955 memoplen 8
We saw an instruction at vdso vaddr 0x74b with len 7 displacement -5954 memoplen 8
We saw an instruction at vdso vaddr 0x771 with len 6 displacement -10103 memoplen 4
We saw an instruction at vdso vaddr 0x777 with len 7 displacement -10110 memoplen 8
We saw an instruction at vdso vaddr 0x784 with len 7 displacement -10094 memoplen 1
We saw an instruction at vdso vaddr 0x7a0 with len 7 displacement -10123 memoplen 1
We saw an instruction at vdso vaddr 0x7a7 with len 7 displacement -10150 memoplen 8
We saw an instruction at vdso vaddr 0x7c2 with len 6 displacement -10160 memoplen 4
We saw an instruction at vdso vaddr 0x7d7 with len 7 displacement -10190 memoplen 8
We saw an instruction at vdso vaddr 0x7fc with len 6 displacement -13978 memoplen 4
We saw an instruction at vdso vaddr 0x860 with len 7 displacement -14311 memoplen 8
We saw an instruction at vdso vaddr 0x8b1 with len 6 displacement -14167 memoplen 4
We saw an instruction at vdso vaddr 0x8bb with len 6 displacement -14173 memoplen 4
We saw an instruction at vdso vaddr 0x8d0 with len 7 displacement -14391 memoplen 8
We saw an instruction at vdso vaddr 0x90c with len 7 displacement -14483 memoplen 8
We saw an instruction at vdso vaddr 0x948 with len 7 displacement -14303 memoplen 8
We saw an instruction at vdso vaddr 0x959 with len 7 displacement -14560 memoplen 8
Decoding 28 bytes from vaddr 0xa20
TRACE_SYSCALLS_TRACE_FD is 1, fd 1 is open; outputting traces there.
trace-syscalls-ld: jumping to system ld.so entry point 0x555555557100 with rsp 0x7ffdf7320020
== 916108 == > 0x555555574089 ((unknown)+0x0) brk(0, 0x55555557be6e, 0x5f, 0x55555557be6e, 0x555555583e88, 0x1c)
== 916108 == > 0x555555572d33 ((unknown)+0x0) arch_prctl(0x3001, 0x7ffdf731ff50, 0x555555572410, 0x55555557a8b8, 0x3, 0x555555573101)
== 916108 == > 0x5555555750a6 ((unknown)+0x0) openat(0xffffff9c, 0x555555585190, 0x80000, 0, 0x80000, 0x555555585190)
== 916108 == > 0x555555575166 ((unknown)+0x0) read(0x3, 0x7ffdf731f8d8, 0x340, 0, 0x80000, 0x555555585190)
== 916108 == > 0x55555557519c ((unknown)+0x0) pread64(0x3, 0x7ffdf731f7e0, 0x20, 0x338, 0x80000, 0x7ffdf731f918)
== 916108 == > 0x55555557519c ((unknown)+0x0) pread64(0x3, 0x7ffdf731f790, 0x44, 0x358, 0x80000, 0x7ffdf731f918)
== 916108 == > 0x55555557519c ((unknown)+0x0) pread64(0x3, 0x7ffdf731f400, 0x20, 0x338, 0x3, 0x338)
== 916108 == > 0x55555557519c ((unknown)+0x0) pread64(0x3, 0x7ffdf731f3e0, 0x44, 0x358, 0x3, 0x358)
== 916108 == > 0x5555555752c4 ((unknown)+0x0) mmap(0, 0xa218, 0x1, 0x802, 0x3, 0)
== 916108 == > 0x555555575379 ((unknown)+0x0) mprotect(0x7fb67579d000, 0x7000, 0, 0x802, 0x3, 0)
== 916108 == > 0x5555555752c4 ((unknown)+0x0) mmap(0x7fb67579d000, 0x4000, 0x5, 0x812, 0x3, 0x2000)
== 916108 == > 0x5555555752c4 ((unknown)+0x0) mmap(0x7fb6757a1000, 0x2000, 0x1, 0x812, 0x3, 0x6000)
== 916108 == > 0x5555555752c4 ((unknown)+0x0) mmap(0x7fb6757a4000, 0x2000, 0x3, 0x812, 0x3, 0x8000)
== 916108 == > 0x555555574f69 ((unknown)+0x0) close(0x3, 0x29, 0x20000000, 0x7fb67579b000, 0xeffffef5, 0x70000022)
== 916108 == > 0x5555555753a9 ((unknown)+0x0) uname(0x7ffdf731fab0, 0x7fb6757a6120, 0x54, 0xffff80498a859b4c, 0x54, 0x55555557a9d0)
== 916108 == > 0x555555574f39 (trace-syscalls-ld.so+0x1ef39) 21(0x55555557ead0, 0x4, 0x555555556270, 0x10, 0, 0)
== 916108 == > 0x5555555750a6 (trace-syscalls-ld.so+0x1f0a6) openat(0xffffff9c, 0x55555557bbb3, 0x80000, 0, 0x80000, 0x55555557bbb3)
== 916108 == > 0x555555574e77 (trace-syscalls-ld.so+0x1ee77) fstat(0x3, 0x7ffdf731f0b0, 0x7ffdf731f0b0, 0, 0x1, 0x55555557bbb3)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0, 0x209ba, 0x1, 0x2, 0x3, 0)
== 916108 == > 0x555555574f69 (trace-syscalls-ld.so+0x1ef69) close(0x3, 0x209ba, 0x1, 0x2, 0x3, 0)
== 916108 == > 0x5555555750a6 (trace-syscalls-ld.so+0x1f0a6) openat(0xffffff9c, 0x555555585f70, 0x80000, 0, 0x80000, 0x555555585f70)
== 916108 == > 0x555555575166 (trace-syscalls-ld.so+0x1f166) read(0x3, 0x7ffdf731f258, 0x340, 0, 0x80000, 0x555555585f70)
== 916108 == > 0x55555557519c (trace-syscalls-ld.so+0x1f19c) pread64(0x3, 0x7ffdf731ee70, 0x310, 0x40, 0x7ffdf731ee70, 0x555555585f70)
== 916108 == > 0x55555557519c (trace-syscalls-ld.so+0x1f19c) pread64(0x3, 0x7ffdf731ee40, 0x20, 0x350, 0x7ffdf731ee70, 0)
== 916108 == > 0x55555557519c (trace-syscalls-ld.so+0x1f19c) pread64(0x3, 0x7ffdf731edf0, 0x44, 0x370, 0x7ffdf731ee70, 0)
== 916108 == > 0x555555574e77 (trace-syscalls-ld.so+0x1ee77) fstat(0x3, 0x7ffdf731f100, 0x7ffdf731f100, 0x370, 0x1, 0x5555555851a0)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0, 0x2000, 0x3, 0x22, 0xffffffff, 0)
== 916108 == > 0x55555557519c (trace-syscalls-ld.so+0x1f19c) pread64(0x3, 0x7ffdf731ed50, 0x310, 0x40, 0xffff, 0x7fb675778480)
== 916108 == > 0x55555557519c (trace-syscalls-ld.so+0x1f19c) pread64(0x3, 0x7ffdf731ea30, 0x20, 0x350, 0x3, 0x350)
== 916108 == > 0x55555557519c (trace-syscalls-ld.so+0x1f19c) pread64(0x3, 0x7ffdf731ea10, 0x44, 0x370, 0x3, 0x370)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0, 0x1f14d8, 0x1, 0x802, 0x3, 0)
== 916108 == > 0x555555575379 (trace-syscalls-ld.so+0x1f379) mprotect(0x7fb6755ab000, 0x1c3000, 0, 0x802, 0x3, 0)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0x7fb6755ab000, 0x178000, 0x5, 0x812, 0x3, 0x25000)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0x7fb675723000, 0x4a000, 0x1, 0x812, 0x3, 0x19d000)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0x7fb67576e000, 0x6000, 0x3, 0x812, 0x3, 0x1e7000)
== 916108 == > 0x5555555752c4 (trace-syscalls-ld.so+0x1f2c4) mmap(0x7fb675774000, 0x34d8, 0x3, 0x32, 0xffffffff, 0)
== 916108 == > 0x555555574f69 (trace-syscalls-ld.so+0x1ef69) close(0x3, 0x29, 0, 0x7fb675586000, 0xeffffef5, 0x70000022)
== 916108 == > 0x555555557cce (trace-syscalls-ld.so+0x1cce) arch_prctl(0x1002, 0x7fb675779540, 0xffff80498a886190, 0x40, 0x90, 0x1)
== 916108 == > 0x555555575379 (trace-syscalls-ld.so+0x1f379) mprotect(0x7fb67576e000, 0x3000, 0x1, 0x5555555835a0, 0x7fb675714a60, 0x555555556670)
== 916108 == > 0x555555575379 (trace-syscalls-ld.so+0x1f379) mprotect(0x7fb6757a4000, 0x1000, 0x1, 0x7fb6757a4e18, 0, 0x6)
== 916108 == > 0x555555575379 (trace-syscalls-ld.so+0x1f379) mprotect(0x555555583000, 0x1000, 0x1, 0x555555583f78, 0, 0x7fb67558e790)
== 916108 == > 0x555555575349 (trace-syscalls-ld.so+0x1f349) munmap(0x7fb67577a000, 0x209ba, 0x902f700000000, 0x555555583f78, 0, 0x7fb67558e790)
== 916108 == > 0x7fb67566c2c4 (/lib/x86_64-linux-gnu/libc.so.6+0xe62c4) exit_group(0, 0x3c, 0, 0x2, 0xe7, 0xffffffffffffff80)

[stephenrkell]

Glad to hear! I'll close this issue now, but do raise another if you run into more problems.