Segmentation fault / null pointer exception

[Quuxplusone]

Bugzilla Link	PR50182
Status	NEW
Importance	P normal
Reported by	Marcel Swidersky (marcel.swidersky@stud.tu-darmstadt.de)
Reported on	2021-04-30 09:08:18 -0700
Last modified on	2021-05-01 11:37:42 -0700
Version	trunk
Hardware	PC Linux
CC	dblaikie@gmail.com, llvm-bugs@lists.llvm.org
Fixed by commit(s)
Attachments	minimal_example.tar.gz (934 bytes, application/gzip)
Blocks
Blocked by
See also

Created attachment 24816
A minimal example program that causes the crash + Makefile that generates the
three cases I described

When compiling one of the CFI-showcase programs (attached) with CFI
instrumentation and recovery a nullpointer exception in llvm-symbolizer occurs.

The program is not clean by design - this is the point of a CFI showcase -  but
it runs fine without any instrumentation enabled.

It also runs fine with instrumentation enabled (-fsanitize=cfi-icall) - the
violation is reported and the program aborts gracefully.

However, with instrumentation (-fsanitize=cfi-icall) AND recovery (-fsanitize-
recover=all) enabled, the program does not resume execution after the CFI
violation has been reported. A nullpointer exception causes a crash.

All other CFI showcase programs resume correctly, only this one case causes a
crash.

Expected behavior:
$ ./cfi_skip_entrypoint
cfi_skip_entrypoint.c:35:15: runtime error: control flow integrity check for
type 'int (int)' failed during indirect function call
CFI ensures control flow only transfers to potentially valid destinations
In not_entry_point: (0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you can read this, execution has resumed!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

>>> The program should report the violation and continue execution

Actual behavior:
$ ./cfi_skip_entrypoint
cfi_skip_entrypoint.c:35:15: runtime error: control flow integrity check for
type 'int (int)' failed during indirect function call
(/mnt/clang-cfi-showcase/cfi_skip_entrypoint+0x423c40): note: __libc_csu_init
defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
cfi_skip_entrypoint.c:35:15 in
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==23263==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address
0x000000000000 (pc 0x000000423c71 bp 0x000000086ce8 sp 0x7ffe258018d8 T23263)
==23263==The signal is caused by a READ memory access.
==23263==Hint: address points to the zero page.
    #0 0x423c71 in __libc_csu_init (/mnt/clang-cfi-showcase/cfi_skip_entrypoint+0x423c71)
    #1 0x423c2f  (/mnt/clang-cfi-showcase/cfi_skip_entrypoint+0x423c2f)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV (/mnt/clang-cfi-
showcase/cfi_skip_entrypoint+0x423c71) in __libc_csu_init
==23263==ABORTING

>>> The program reports the violation but cannot resume because of a
nullpointer exception

This is what I got after debugging it in gdb:
--------------------------------------------------------------------------------
Thread 2.1 "llvm-symbolizer" received signal SIGSEGV, Segmentation fault.

Run till exit from #0  __libc_start_main (main=0x423ad0 <main>, argc=1,
argv=0x7fffffffe758, init=0x423c30 <__libc_csu_init>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffe748) at ../csu/libc-start.c:268
cfi_skip_entrypoint.c:35:15: runtime error: control flow integrity check for
type 'int (int)' failed during indirect function call

>>> CFI violation is reported correctly <<<

[Attaching after Thread 0x7ffff78bdb80 (LWP 35794) fork to child process 35828]
[New inferior 2 (process 35828)]
[Detaching after fork from parent process 35794]
[Inferior 1 (process 35794) detached]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
process 35828 is executing new program: /usr/lib/llvm-11/bin/llvm-symbolizer

>>> Execution resumes, llvm-symbolizer is called <<<

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Thread 2.1 "llvm-symbolizer" received signal SIGSEGV, Segmentation fault.
--------------------------------------------------------------------------------

[Quuxplusone]

Attached minimal_example.tar.gz (934 bytes, application/gzip): A minimal example program that causes the crash + Makefile that generates the three cases I described

[Quuxplusone]

You might be able to reduce the test case a bit if you can figure out what's being passed to llvm-symbolizer and run just that - llvm-symbolizer shouldn't crash regardless (so that's at least one bug) - though it's possible it's being passed bogus/things from the sanitizer too (could be another bug).

[Quuxplusone]

(In reply to David Blaikie from comment #1)
> You might be able to reduce the test case a bit if you can figure out what's
> being passed to llvm-symbolizer and run just that - llvm-symbolizer
> shouldn't crash regardless (so that's at least one bug) - though it's
> possible it's being passed bogus/things from the sanitizer too (could be
> another bug).

I'm an undergrad student, so my skills are limited. But using strace I think I
found out what is being passed to llvm-symbolizer:

...
14829 execve("/mnt/llvm-project/build/bin/llvm-symbolizer", ["/mnt/llvm-
project/build/bin/llvm-symbolizer", "--inlines", "--default-arch=x86_64"],
["MAIL=/var/mail/marcel", "USER=marcel", "XDG_SEAT=seat0",
"XDG_SESSION_TYPE=tty", "SHLVL=2", "HOME=/home/marcel", "MOTD_SHOWN=pam",
"OLDPWD=/home/marcel", "HUSHLOGIN=FALSE", "LOGNAME=marcel",
"JOURNAL_STREAM=8:15158", "_=/usr/bin/strace", "XDG_SESSION_CLASS=user",
"TERM=xterm-kitty", "XDG_SESSION_ID=1", "WINDOWPATH=1", "PATH=/mnt/llvm-
project/build/bin:/home/marcel/.local/share/texlive/2021/bin/x86_64-
linux:/home/marcel/.local/bin:usr/local/bin:/usr/sbin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games",
"INVOCATION_ID=3e7ecf722c854e1f85abd397636b6052",
"XDG_RUNTIME_DIR=/run/user/1000", "DISPLAY=:0", "LANG=de_DE.UTF-8",
"XAUTHORITY=/home/marcel/.Xauthority", "SHELL=/bin/zsh", "XDG_VTNR=1",
"PWD=/mnt/clang-cfi-showcase", "SWM_STARTED=YES",
"LD_PRELOAD=/usr/lib/spectrwm/libswmhack.so.0.0", "_SWM_WS=0",
"_SWM_PID=12029", "KITTY_WINDOW_ID=1", "WINDOWID=71303182",
"COLORTERM=truecolor", "LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:",
"MANPATH=/usr/local/man:/home/marcel/.local/share/texlive/2021/texmf-
dist/doc/man:", "INFOPATH=/home/marcel/.local/share/texlive/2021/texmf-
dist/doc/info:", "EDITOR=vim", "VISUAL=vim", "MANPAGER=/bin/zsh -c \"vim -MRn -
c \\\"set buftype=nofile showtabline=0 ft=man ts=8 nomod nolist
norelativenumber nonu noma\\\" -c \\\"normal L\\\" -c \\\"nmap q
:qa<CR>\\\"</dev/tty <(col -b)\"",
"NNN_BMS=d:~/Dokumente;u:~/Dokumente/Uni/;D:~/Downloads;t:/tmp;m:~/Musik",
"NNN_COLORS=6145", "NNN_PLUG=m:nmount", "LS_COLORS="]) = 0
...

But I don't know if that is helpful. I wrote a minimal C program that calls
this execve and compiled it with the same arguments and compiler as the other
program and it does *not* crash.

[Quuxplusone]

(In reply to Marcel Swidersky from comment #2)
> (In reply to David Blaikie from comment #1)
> > You might be able to reduce the test case a bit if you can figure out what's
> > being passed to llvm-symbolizer and run just that - llvm-symbolizer
> > shouldn't crash regardless (so that's at least one bug) - though it's
> > possible it's being passed bogus/things from the sanitizer too (could be
> > another bug).
>
> I'm an undergrad student, so my skills are limited. But using strace I think
> I found out what is being passed to llvm-symbolizer:
>
> ...
> 14829 execve("/mnt/llvm-project/build/bin/llvm-symbolizer",
> ["/mnt/llvm-project/build/bin/llvm-symbolizer", "--inlines",
> "--default-arch=x86_64"], ["MAIL=/var/mail/marcel", "USER=marcel",
> "XDG_SEAT=seat0", "XDG_SESSION_TYPE=tty", "SHLVL=2", "HOME=/home/marcel",
> "MOTD_SHOWN=pam", "OLDPWD=/home/marcel", "HUSHLOGIN=FALSE",
> "LOGNAME=marcel", "JOURNAL_STREAM=8:15158", "_=/usr/bin/strace",
> "XDG_SESSION_CLASS=user", "TERM=xterm-kitty", "XDG_SESSION_ID=1",
> "WINDOWPATH=1",
> "PATH=/mnt/llvm-project/build/bin:/home/marcel/.local/share/texlive/2021/bin/
> x86_64-linux:/home/marcel/.local/bin:usr/local/bin:/usr/sbin/:/usr/local/bin:
> /usr/bin:/bin:/usr/local/games:/usr/games",
> "INVOCATION_ID=3e7ecf722c854e1f85abd397636b6052",
> "XDG_RUNTIME_DIR=/run/user/1000", "DISPLAY=:0", "LANG=de_DE.UTF-8",
> "XAUTHORITY=/home/marcel/.Xauthority", "SHELL=/bin/zsh", "XDG_VTNR=1",
> "PWD=/mnt/clang-cfi-showcase", "SWM_STARTED=YES",
> "LD_PRELOAD=/usr/lib/spectrwm/libswmhack.so.0.0", "_SWM_WS=0",
> "_SWM_PID=12029", "KITTY_WINDOW_ID=1", "WINDOWID=71303182",
> "COLORTERM=truecolor", "LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:",
> "MANPATH=/usr/local/man:/home/marcel/.local/share/texlive/2021/texmf-dist/
> doc/man:",
> "INFOPATH=/home/marcel/.local/share/texlive/2021/texmf-dist/doc/info:",
> "EDITOR=vim", "VISUAL=vim", "MANPAGER=/bin/zsh -c \"vim -MRn -c \\\"set
> buftype=nofile showtabline=0 ft=man ts=8 nomod nolist norelativenumber nonu
> noma\\\" -c \\\"normal L\\\" -c \\\"nmap q :qa<CR>\\\"</dev/tty <(col
> -b)\"",
> "NNN_BMS=d:~/Dokumente;u:~/Dokumente/Uni/;D:~/Downloads;t:/tmp;m:~/Musik",
> "NNN_COLORS=6145", "NNN_PLUG=m:nmount", "LS_COLORS="]) = 0
> ...
>
> But I don't know if that is helpful. I wrote a minimal C program that calls
> this execve and compiled it with the same arguments and compiler as the
> other program and it does *not* crash.

The other thing that probably happens is that some addresses are fed to the
standard input of llvm-symbolizer - it might be that reading/trying to process
one of those addresses is what's causing the crash. So you'd probably need to
discover/reproduce that standard input too.

[Quuxplusone]

(In reply to David Blaikie from comment #3)
> (In reply to Marcel Swidersky from comment #2)
> > (In reply to David Blaikie from comment #1)
> > > You might be able to reduce the test case a bit if you can figure out
what's
> > > being passed to llvm-symbolizer and run just that - llvm-symbolizer
> > > shouldn't crash regardless (so that's at least one bug) - though it's
> > > possible it's being passed bogus/things from the sanitizer too (could be
> > > another bug).
> >
> > I'm an undergrad student, so my skills are limited. But using strace I think
> > I found out what is being passed to llvm-symbolizer:
> >
> > ...
> > 14829 execve("/mnt/llvm-project/build/bin/llvm-symbolizer",
> > ["/mnt/llvm-project/build/bin/llvm-symbolizer", "--inlines",
> > "--default-arch=x86_64"], ["MAIL=/var/mail/marcel", "USER=marcel",
> > "XDG_SEAT=seat0", "XDG_SESSION_TYPE=tty", "SHLVL=2", "HOME=/home/marcel",
> > "MOTD_SHOWN=pam", "OLDPWD=/home/marcel", "HUSHLOGIN=FALSE",
> > "LOGNAME=marcel", "JOURNAL_STREAM=8:15158", "_=/usr/bin/strace",
> > "XDG_SESSION_CLASS=user", "TERM=xterm-kitty", "XDG_SESSION_ID=1",
> > "WINDOWPATH=1",
> > "PATH=/mnt/llvm-
project/build/bin:/home/marcel/.local/share/texlive/2021/bin/
> > x86_64-
linux:/home/marcel/.local/bin:usr/local/bin:/usr/sbin/:/usr/local/bin:
> > /usr/bin:/bin:/usr/local/games:/usr/games",
> > "INVOCATION_ID=3e7ecf722c854e1f85abd397636b6052",
> > "XDG_RUNTIME_DIR=/run/user/1000", "DISPLAY=:0", "LANG=de_DE.UTF-8",
> > "XAUTHORITY=/home/marcel/.Xauthority", "SHELL=/bin/zsh", "XDG_VTNR=1",
> > "PWD=/mnt/clang-cfi-showcase", "SWM_STARTED=YES",
> > "LD_PRELOAD=/usr/lib/spectrwm/libswmhack.so.0.0", "_SWM_WS=0",
> > "_SWM_PID=12029", "KITTY_WINDOW_ID=1", "WINDOWID=71303182",
> > "COLORTERM=truecolor", "LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:",
> > "MANPATH=/usr/local/man:/home/marcel/.local/share/texlive/2021/texmf-dist/
> > doc/man:",
> > "INFOPATH=/home/marcel/.local/share/texlive/2021/texmf-dist/doc/info:",
> > "EDITOR=vim", "VISUAL=vim", "MANPAGER=/bin/zsh -c \"vim -MRn -c \\\"set
> > buftype=nofile showtabline=0 ft=man ts=8 nomod nolist norelativenumber nonu
> > noma\\\" -c \\\"normal L\\\" -c \\\"nmap q :qa<CR>\\\"</dev/tty <(col
> > -b)\"",
> > "NNN_BMS=d:~/Dokumente;u:~/Dokumente/Uni/;D:~/Downloads;t:/tmp;m:~/Musik",
> > "NNN_COLORS=6145", "NNN_PLUG=m:nmount", "LS_COLORS="]) = 0
> > ...
> >
> > But I don't know if that is helpful. I wrote a minimal C program that calls
> > this execve and compiled it with the same arguments and compiler as the
> > other program and it does *not* crash.
>
> The other thing that probably happens is that some addresses are fed to the
> standard input of llvm-symbolizer - it might be that reading/trying to
> process one of those addresses is what's causing the crash. So you'd
> probably need to discover/reproduce that standard input too.

I used a wrapper script from here:

and found this STDIN:

CODE "/mnt/clang-cfi-showcase/cfi_skip_entrypoint" 0x425691
CODE "/mnt/clang-cfi-showcase/cfi_skip_entrypoint" 0x42564f
CODE "/mnt/clang-cfi-showcase/cfi_skip_entrypoint" 0x425691

to generate this STDOUT:

__libc_csu_init
??:0:0

??
??:0:0

__libc_csu_init
??:0:0

But again, no crash when I run cat /tmp/logname-stdin | /mnt/llvm-
project/build/bin/llvm-symbolizer --inlines --default-arch=x86_64

Perhaps the symbolizer is not the problem after all and gdb wrongly reported
that there was the origin of the nullpointer exception? I'm not skilled enough
to go on from here.

[Quuxplusone]

Fair enough - thanks for giving it a go!