snare / voltron

A hacky debugger UI for hackers
MIT License
6.18k stars 414 forks source link

Various segfaults using gdb with current version of voltron #203

Open Flowm opened 7 years ago

Flowm commented 7 years ago

When using gdb with voltron (cd11d2f) on Debian 8, gdb will segfault (or crash with other reasons) after the first few step commands when a voltron view is attached.

Stacktrace and overall problem seems similar to #81.

Issue seems to be independent of debugged program. Basic steps:

# cat test.c
#include <stdio.h>
#include <unistd.h>

int main() {
        while (1) {
                printf("x\n");
                sleep(1);
        }

        return 0;
}

# gcc test.c -o test
# voltron v reg # somwhere else
# gdb test
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Voltron loaded.
Reading symbols from test...(no debugging symbols found)...done.
(gdb) b main
Breakpoint 1 at 0x40054a
(gdb) r
Starting program: /home/vagrant/dev/vdbg/test

Breakpoint 1, 0x000000000040054a in main ()
(gdb) si
0x000000000040054f in main ()
(gdb)
0x0000000000400410 in puts@plt ()
(gdb)
0x0000000000400416 in puts@plt ()
(gdb)
0x000000000040041b in puts@plt ()
(gdb)
0x0000000000400400 in ?? ()
(gdb)
0x0000000000400406 in ?? ()
(gdb)
_dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:34
34      ../sysdeps/x86_64/dl-trampoline.S: No such file or directory.
(gdb)
36      in ../sysdeps/x86_64/dl-trampoline.S
(gdb) /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/cleanups.c:92: internal-error: make_my_cleanup2: Assertion `old_chain != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) [1]    1383 segmentation fault  gdb test
...
(gdb)
_dl_fixup (l=0x7ffff7ffe1a8, reloc_arg=0) at ../elf/dl-runtime.c:66
66      ../elf/dl-runtime.c: No such file or directory.
(gdb)
*** Error in `gdb': double free or corruption (fasttop): 0x00007fbbec013ad0 ***
# gdb $(which gdb) core -ex bt
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Voltron loaded.
Reading symbols from /usr/bin/gdb...Reading symbols from /usr/lib/debug/.build-id/d5/a96c0efad4fecb3cf1e09d3b2399d14246d773.debug...done.
done.
[New LWP 3093]
[New LWP 3097]
[New LWP 3091]
[New LWP 3109]
[New LWP 3110]
[New LWP 3078]
[New LWP 3092]
[New LWP 3105]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `gdb test'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fbbfbb40067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  0x00007fbbfbb40067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fbbfbb41448 in __GI_abort () at abort.c:89
#2  0x0000000000655e26 in dump_core () at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/utils.c:635
#3  0x00000000006582d5 in internal_vproblem (problem=problem@entry=0xb81bc0 <internal_error_problem>, file=<optimized out>, line=<optimized out>, fmt=<optimized out>, ap=ap@entry=0x7fbbf617d7a0)
    at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/utils.c:794
#4  0x0000000000658359 in internal_verror (file=<optimized out>, line=<optimized out>, fmt=<optimized out>, ap=ap@entry=0x7fbbf617d7a0) at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/utils.c:820
#5  0x000000000065840f in internal_error (file=file@entry=0x79c148 "/build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/thread.c", line=line@entry=628, string=<optimized out>)
    at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/utils.c:830
#6  0x000000000058ff8c in is_thread_state (ptid=..., state=THREAD_EXITED) at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/thread.c:628
#7  0x000000000065d456 in has_stack_frames () at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/frame.c:1494
#8  0x000000000065dfd9 in deprecated_safe_get_selected_frame () at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/frame.c:1541
#9  0x0000000000653f59 in check_frame_language_change () at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/top.c:355
#10 0x000000000065410e in execute_command (p=<optimized out>, p@entry=0x7fbbec0096d0 "info program", from_tty=from_tty@entry=0) at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/top.c:486
#11 0x000000000065430b in execute_command_to_string (p=p@entry=0x7fbbec0096d0 "info program", from_tty=from_tty@entry=0) at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/top.c:530
#12 0x00000000004e204c in execute_gdb_command (self=<optimized out>, args=<optimized out>, kw=<optimized out>) at /build/gdb-2hEJVN/gdb-7.7.1+dfsg/gdb/python/python.c:559
#13 0x00007fbbfc899da4 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#21 0x00007fbbfc85b40d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#22 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#23 0x00007fbbfc896907 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#24 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#25 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#26 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#27 0x00007fbbfc85b40d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#28 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#29 0x00007fbbfc896907 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#30 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#31 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#32 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#33 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#34 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#35 0x00007fbbfc85b40d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#36 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#37 0x00007fbbfc896907 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#38 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#39 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#40 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#41 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#42 0x00007fbbfc85b32c in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#43 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#44 0x00007fbbfc8a60e5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#45 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#46 0x00007fbbfc7da335 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#47 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#48 0x00007fbbfc897442 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#49 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#50 0x00007fbbfc85b32c in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#51 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#52 0x00007fbbfc8a60e5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#53 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#54 0x00007fbbfc7da335 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#55 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#56 0x00007fbbfc897442 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#57 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#58 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#59 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#60 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#61 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#62 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#63 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#64 0x00007fbbfc85b32c in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#65 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#66 0x00007fbbfc8a60e5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#67 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#68 0x00007fbbfc897442 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#69 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#70 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#71 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#72 0x00007fbbfc85b32c in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#73 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#74 0x00007fbbfc8a60e5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#75 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#76 0x00007fbbfc7b9d8d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#77 0x00007fbbfc7dbd2f in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#78 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#79 0x00007fbbfc897442 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#80 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#81 0x00007fbbfc85b32c in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#82 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#83 0x00007fbbfc896907 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#84 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#85 0x00007fbbfc899171 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#86 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#87 0x00007fbbfc85b40d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#88 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#89 0x00007fbbfc896907 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#90 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#91 0x00007fbbfc89927d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#92 0x00007fbbfc90c190 in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#93 0x00007fbbfc85b32c in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#94 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#95 0x00007fbbfc8a60e5 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#96 0x00007fbbfc863be3 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#97 0x00007fbbfc90b6e7 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#98 0x00007fbbfc783dc2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#99 0x00007fbbfc50d064 in start_thread (arg=0x7fbbf6182700) at pthread_create.c:309
#100 0x00007fbbfbbf362d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

Issue also occurs with minimal .gdbinit (only containing source /home/vagrant/.local/lib/python2.7/site-packages/voltron/entry.py)

Gdb version Debian 8:

# gdb -v
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
# gdb -batch -q --nx -ex 'pi import platform; print(".".join(platform.python_version_tuple()[:2]))'
2.7

Identical behavior can be observed with gdb + voltron on Ubuntu 16.04:

# gdb -v
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
# gdb -batch -q --nx -ex 'pi import platform; print(".".join(platform.python_version_tuple()[:2]))'
3.5

After going back to the the latest released voltron version (v1.7) everything seems to be working fine.

A git bisect of the commits after v1.7 returned the following commit introducing the issue:

vagrant@pwn ~/dev/voltron ±42306ff » git bisect good
8691b44c5977d2d3e48231c7a7274c7dfaef367e is the first bad commit
commit 8691b44c5977d2d3e48231c7a7274c7dfaef367e
Author: snare <snare@ho.ax>
Date:   Wed Oct 19 22:27:29 2016 +1100

    Convert GDB adaptor to use post_event for magical async

vagrant@pwn ~/dev/voltron » git lgl
* 8691b44 (refs/bisect/bad) Convert GDB adaptor to use post_event for magical async (7 months ago) <snare>
* 42306ff (refs/bisect/good-42306ffae66c26ed9c6ac608b5fa86984e38a2ec) Yer an idiot, 'arry (7 months ago) <snare>
* c0c443c Disabling OS X Travis test until it can be more reliable (7 months ago) <snare>
* 31371e9 (refs/bisect/good-31371e9075674568b76465169ba4de357daf1637) Update script to work properly when GDB and LLDB are both installed, and work properly with LLDB on Linux (7 months ago) <snare>
* 089bca2 Use install script for testing (7 months ago) <snare>
* 776a84c (tag: v0.1.7, refs/bisect/good-776a84cb0eff530f43ed96f0893e957f4cd5fbf1) Version bump (8 months ago) <snare>

As a workaround reverting the changes to voltron/plugins/debugger/dbg_gdb.py since v1.7 seems to work fine. (https://github.com/Flowm/voltron/commits/gdb-no-segv)

Ichiji001 commented 6 years ago

I have this exact same issue. I'm using Ubuntu 18.04 gdb compiled with python3

snare commented 6 years ago

I'll investigate. Using post_event was actually to avoid issues related to multiple threads doing concurrent API calls, which was causing crashes in previous versions.

vitapluvia commented 5 years ago

Also having issues with Voltron a lot -- I'd love to use it, there are so many nice features and it improves the experience of gdb a ton, but it's been unusable with how many crashes come up. The main reproduction I've been seeing is when rerunning a binary, ex.:

This has been on multiple gdb versions & Ubuntu versions as well.

Mithi83 commented 4 years ago

I've run into this problem as well on

Reverting commit 8691b44c5977d2d3e48231c7a7274c7dfaef367e (with some educated guesses on how to fix merge conflicts) seems to work better, though I have only tried a very small executable for a short time.

@snare: Did you find any clues during your investigation?

the-exodus commented 4 years ago

Pulling the branch from @Flowm s fork everything works as long as I don't attempt to follow children on forks. I will start poking at this (I assume it's a synchronization issue), but would be good to know if @snare has found any clues first, as I'm not familiar with either GDB internals or the Voltron code base.

Mithi83 commented 4 years ago

I tried fiddling around with the original code a bit and I think I found a synchronization issue there.

The decorator @post_event is supposed to synchronize access to gdb and probably does a decent job at that. It is however not properly used in all instances of access to gdb from what I can see. The API of GDBAdaptor lists many functions with multiple decorators, most in this particular order (@validate_busy, @validate_target, @post_event). Thus, the actions within the outer decorators @validate_busy and @validate_target are not synchronized. These functions come from dbg.py and internally call target_is_busy resp. target_is_valid, which in turn call self._target. The _target function comes from dbg_gdb.py and directly accesses gdb without the synchronization of the @post_event decorator, creating a potential race condition within any GDBAdaptor API function that has @validate_busy or @validate_target decorators.

For a quick test I changed all occurrences of self._target within dbg.py to self.target, which is a wrapper around the internal self._target plus the synchronization of the @post_event decorator. It seems to work at least a lot better. I'm not sure if this is a proper fix though, since I can't foresee the impact of this change on other debugger plugins. Maybe someone with more insight into the code can take it from here.