search

pwndbg / pwndbg

Exploit Development and Reverse Engineering with GDB Made Easy
https://pwndbg.re/
MIT License
7.48k stars 878 forks source link

Remote debugging fails with 'g' packet reply too long #619

Closed nop-90 closed 2 years ago

nop-90 commented 5 years ago

Hi,

While debugging a program on an ARM virtual machine (Qemu) using remote debugging, GDB tried to follow a program fork as I explicitly configured with follow-fork-mode but pwndbg failed to properly parse the contents of the gdb server response for the command show osabi. It received 328 bytes instead of 168 bytes. I tried on vanilla GDB and it didn't failed.

Thanks,

Description

Continuing.
[Attaching after Thread 1263.1263 fork to child Thread 1264.1264]
[New inferior 2 (process 1264)]
[Detaching after fork from parent process 1263]
[Inferior 1 (process 1263) detached]
process 1264 is executing new program: /bin/busybox
Traceback (most recent call last):
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/abi.py", line 138, in update
    abi = gdb.execute('show osabi', to_string=True).split('\n')[0]
gdb.error: Remote 'g' packet reply is too long (expected 168 bytes, got 328 bytes): 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000506ddd7e00000000e097f676100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

If that is an issue, you can report it on https://github.com/pwndbg/pwndbg/issues
(Please don't forget to search if it hasn't been reported before)
To generate the report and open a browser, you may run `bugreport --run-browser`
PS: Pull requests are welcome
Traceback (most recent call last):
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 238, in _start_newobjfile
    gdb.events.start.on_new_objfile()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 60, in on_new_objfile
    function()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 173, in caller
    raise e
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/abi.py", line 138, in update
    abi = gdb.execute('show osabi', to_string=True).split('\n')[0]
gdb.error: Remote 'g' packet reply is too long (expected 168 bytes, got 328 bytes): 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000506ddd7e00000000e097f676100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

If that is an issue, you can report it on https://github.com/pwndbg/pwndbg/issues
(Please don't forget to search if it hasn't been reported before)
To generate the report and open a browser, you may run `bugreport --run-browser`
PS: Pull requests are welcome
Traceback (most recent call last):
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 173, in caller
    raise e
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 238, in _start_newobjfile
    gdb.events.start.on_new_objfile()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 60, in on_new_objfile
    function()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 173, in caller
    raise e
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/abi.py", line 138, in update
    abi = gdb.execute('show osabi', to_string=True).split('\n')[0]
gdb.error: Remote 'g' packet reply is too long (expected 168 bytes, got 328 bytes): 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000506ddd7e00000000e097f676100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Could not check ASLR: Couldn't get personality

Steps to reproduce

Gdb session history: set sysroot rootfs/ set architecture arm set follow-fork-mode child target remote :2324 set exception-verbose on c

My setup

GDB client

Platform: Linux-4.19.0-kali4-amd64-x86_64-with-Kali-kali-rolling-kali-rolling Gdb: 8.2.1 Python: 3.7.3rc1 (default, Mar 13 2019, 11:01:15) [GCC 8.3.0] Pwndbg: 1.1.0 build: 45a624e Capstone: 4.0.1024 Unicorn: 1.0.1 This GDB was configured as follows: configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-auto-load-dir=$debugdir:$datadir/auto-load --with-auto-load-safe-path=$debugdir:$datadir/auto-load --with-expat --with-gdb-datadir=/usr/share/gdb (relocatable) --with-jit-reader-dir=/usr/lib/gdb (relocatable) --without-libunwind-ia64 --with-lzma --with-babeltrace --with-intel-pt --disable-libmcheck --without-mpfr --with-python=/usr (relocatable) --without-guile --with-separate-debug-dir=/usr/lib/debug (relocatable) --with-system-gdbinit=/etc/gdb/gdbinit

GDB server

gdbserver --version GNU gdbserver (GDB) 8.0.1 This gdbserver was configured as "armv5tejl-unknown-linux-gnueabi"

disconnect3d commented 5 years ago

Does this error happen if u do pi gdb.execute('show osabi', to_string=True) when debugging remotely without pwndbg?

nop-90 commented 5 years ago

I should precise that I use gdb-multiarch

Here's what it does :

(gdb) pi gdb.execute('show osabi', to_string=True)
'The current OS ABI is "auto" (currently "GNU/Linux").\nThe default OS ABI is "GNU/Linux".\n'
disconnect3d commented 5 years ago

Hmm... we should probably have some mode to track all the GDB API calls. Maybe we should mock all gdb functions when initializing pwndbg to add some logging?

With sth like this:

import gdb

def logme(func):
    @functools.wraps(func)
    def _decorator(*a, **kw):
        print("Calling {}({}, {})".format(func.__name__, a, kw))
        result = func(*a, **kw)
        print("Result %s" % result)
        return result
    return _decorator

for attr in dir(gdb):
    val = getattr(gdb, attr)
    if isinstance(val, function):
        setattr(gdb, attr, logme(val))

@nop-90 Could you try to hack this in and show a log? (And maybe send a PR adding this with an if 0 so it is easy to add it in for debugging)

nop-90 commented 5 years ago

I added this into my gdbinit.py with some minor change as isinstance of 'function' didn't work on my python but it didn't work at all. GDB took 3 years to load and crashed with a MemoryError.

import gdb
import functools
import inspect

def logme(func):
    @functools.wraps(func)
    def _decorator(*a, **kw):
        print("Calling {}({}, {})".format(func.__name__, a, kw))
        result = func(*a, **kw)
        print("Result %s" % result)
        return result
    return _decorator

#for attr in dir(gdb):
#    val = getattr(gdb, attr)
#    if inspect.isbuiltin(val):
#        setattr(gdb, attr, logme(val))
setattr(gdb, "execute", logme(getattr(gdb,"execute")))

import pwndbg # isort:skip

I tried to circumvent the problem by just adding the decorator to the execute function of gdb but it made too much logs with 'info file' command results taking all the place. I can give you the entire log if you need it but I need to scrub some paths on it before. The part of the log where the error happens is below but it's not much more informative.

[New inferior 4 (process 1070)]
[Detaching after fork from parent process 1069]
[Inferior 3 (process 1069) detached]
process 1070 is executing new program: /bin/tool
Calling execute(('info file',), {'to_string': True, 'from_tty': False})
Result Symbols from "/home/nop-90/Documents/rootfs/bin/tool".
Remote serial target in gdb-specific protocol:
Debugging a target over a serial line.
    While running this, GDB does not access memory from...
Local exec file:
    `/home/nop-90/Documents/rootfs/bin/tool', file type elf32-littlearm.
    Entry point: 0x8f58
    0x00008174 - 0x00008187 is .interp
    0x00008188 - 0x000081a8 is .note.ABI-tag
    0x000081a8 - 0x00008328 is .hash
    0x00008328 - 0x000084e4 is .gnu.hash
    0x000084e4 - 0x00008874 is .dynsym
    0x00008874 - 0x00008ae8 is .dynstr
    0x00008ae8 - 0x00008b5a is .gnu.version
    0x00008b5c - 0x00008bbc is .gnu.version_r
    0x00008bbc - 0x00008bc4 is .rel.dyn
    0x00008bc4 - 0x00008d24 is .rel.plt
    0x00008d24 - 0x00008d34 is .init
    0x00008d34 - 0x00008f58 is .plt
    0x00008f58 - 0x0000b634 is .text
    0x0000b634 - 0x0000b640 is .fini
    0x0000b640 - 0x0000c1f8 is .rodata
    0x0000c1f8 - 0x0000c200 is .ARM.exidx
    0x0000c200 - 0x0000c204 is .eh_frame
    0x00014ed4 - 0x00014ed8 is .init_array
    0x00014ed8 - 0x00014edc is .fini_array
    0x00014edc - 0x00014ee0 is .jcr
    0x00014ee0 - 0x00015000 is .dynamic
    0x00015000 - 0x000150c0 is .got
    0x000150c0 - 0x000151dc is .data
    0x000151dc - 0x000151f4 is .bss

Calling execute(('show osabi',), {'to_string': True})
Exception occured: Error: Remote 'g' packet reply is too long (expected 168 bytes, got 328 bytes): 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020eadd7e00000000e087f176100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 (<class 'gdb.error'>)

I tried to trace this command 'show osabi' on the network but it seems that it is launched locally on the gdb client and does not send any request to gdbserver. Whereas the error seems to point an incorrect response to a registry dump request and I don't understand how those two operations are linked. Here is a log with set debug remote 1 enabled.

[New inferior 4 (process 1003)]
[Detaching after fork from parent process 1002]
Sending packet: $Hgp3ea.3ea#3f...Packet received: OK
Sending packet: $z0,76fd9e54,4#74...Packet received: OK
Sending packet: $D;3ea#78...Packet received: OK
[Inferior 3 (process 1002) detached]
Sending packet: $Hgp3eb.3eb#41...Packet received: OK
Sending packet: $m76e12130,4#96...Packet received: 0040a0e1
Sending packet: $m76e12130,4#96...Packet received: 0040a0e1
Sending packet: $m76e12130,4#96...Packet received: 0040a0e1
Sending packet: $Z0,76fd9e54,4#54...Packet received: OK
Sending packet: $g#67...Packet received: 0000000000000000000000000000000028c4fe7600000000ea0300007800000038f3a67e0000000000a0ef769cf3a67ee0c3fe7638f3a67e20c8fe76781fe77610000060xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx[144 bytes omitted]
Sending packet: $vCont;c:p3eb.-1#d8...Packet received: T05exec:2f62696e2f6163755f746f6f6c;0b:00000000;0d:209a897e;0f:e027f076;thread:p3eb.3eb;core:0;
process 1003 is executing new program: /bin/tool
Sending packet: $qSymbol::#5b...Packet received: OK
Sending packet: $g#67...Packet received: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000209a897e00000000e027f076100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000[144 bytes omitted]
Traceback (most recent call last):
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/abi.py", line 138, in update
    abi = gdb.execute('show osabi', to_string=True).split('\n')[0]
gdb.error: Remote 'g' packet reply is too long (expected 168 bytes, got 328 bytes): 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000209a897e00000000e027f076100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

I don't know if that's a pwndbg problem but it's seems weird that gdb is not synchronized with it's received packets. Maybe if I trace others function I could see more info but I will see that later

disconnect3d commented 5 years ago

Regarding:

isinstance of 'function' didn't work on my python

https://stackoverflow.com/questions/624926/how-do-i-detect-whether-a-python-variable-is-a-function Probably a callable(obj) is way to go here but isbuiltin might also be okay?

I also wonder if to_string=... or from_tty=... might make some problems here...

Maybe the gdbserver also has some debug flag?

stnevans commented 5 years ago

Are both /bin/tool and whatever you were debugging before /bin/tool started the same architecture? (i.e.) both 32 bit?

From what I saw when reading the gdb source, this error occurs when the architecture gdb believes it is debugging doesn't match with what it actually is. (reference: https://github.com/bminor/binutils-gdb/blob/97dfbaddad2b5acf3dd9f2c3e0135b89bde1cbf8/gdb/remote.c#L8001)

nop-90 commented 5 years ago

Regarding:

isinstance of 'function' didn't work on my python

https://stackoverflow.com/questions/624926/how-do-i-detect-whether-a-python-variable-is-a-function Probably a callable(obj) is way to go here but isbuiltin might also be okay?

I used that at first but it also crashed. I used builtin to reduce the perimeter of the functions affected but it also didn't work.

I also wonder if to_string=... or from_tty=... might make some problems here...

Maybe the gdbserver also has some debug flag?

I will try without this parameter tomorrow

Are both /bin/tool and whatever you were debugging before /bin/tool started the same architecture? (i.e.) both 32 bit?

I'm reversing a firmware so every binary present should use the same architecture and the kernel i'm using to emulate is for ARMv7 (so it shouldn't conflict with AArch64) and I told GDB to set architecture to ARM so it should be aware that those others binaries are also ARM binaries. But it seems there is a conflict of architecture between my gdb-multiarch (kali repo) and the gdbserver that I've added to the firmware. I will try another package of gdb-multiarch to see if there is any difference.

nop-90 commented 5 years ago

GDBserver (with --debug --remote-debug)

putpkt ("$T05exec:2f7573722f6c6f63616c2f7*"2f6367692d62696e2f7365746167656e742e636769;0b:0*"00;0d:f04ad97e;0f:e047fa76;thread:p47f.47f;core:0;#e3"); [noack mode]
handling possible serial event
getpkt ("qSymbol::");  [no ack sent] 
Trying host libthread_db library: libthread_db.so.1.
dlopen failed: /lib//libthread_db.so.1: undefined symbol: ps_pglobal_lookup.
thread_db_load_search returning 0
putpkt ("$OK#9a"); [noack mode]
handling possible serial event
getpkt ("g");  [no ack sent] 
putpkt ("$0*}0*"0f04ad97e0*"00e047fa7610*}0*}0*}0*}0*}0*F#fb"); [noack mode]
handling possible serial event
getpkt ("g");  [no ack sent] 
putpkt ("$0*}0*"0f04ad97e0*"00e047fa7610*}0*}0*}0*}0*}0*F#fb"); [noack mode]
handling possible serial event
getpkt ("g");  [no ack sent] 
putpkt ("$0*}0*"0f04ad97e0*"00e047fa7610*}0*}0*}0*}0*}0*F#fb"); [noack mode]
handling possible serial event
getpkt ("g");  [no ack sent] 
putpkt ("$0*}0*"0f04ad97e0*"00e047fa7610*}0*}0*}0*}0*}0*F#fb"); [noack mode]
handling possible serial event
getpkt ("g");  [no ack sent] 
putpkt ("$0*}0*"0f04ad97e0*"00e047fa7610*}0*}0*}0*}0*}0*F#fb"); [noack mode]
handling possible serial event
getpkt ("qSymbol::");  [no ack sent] 
Trying host libthread_db library: libthread_db.so.1.
dlopen failed: /lib//libthread_db.so.1: undefined symbol: ps_pglobal_lookup.
thread_db_load_search returning 0
putpkt ("$OK#9a"); [noack mode]

GDB client (with debug remote 1)

Sending packet: $vCont;c:p47f.-1#af...Packet received: T05exec:2f7573722f6c6f63616c2f7777772f6367692d62696e2f7365746167656e742e636769;0b:00000000;0d:f04ad97e;0f:e047fa76;thread:p47f.47f;core:0;
process 1151 is executing new program: /usr/local/www/cgi-bin/setagent.cgi
Sending packet: $qSymbol::#5b...Packet received: OK
Sending packet: $g#67...Packet received: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f04ad97e00000000e047fa76100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000[144 bytes omitted]
Traceback (most recent call last):
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/events.py", line 169, in caller
    func()
  File "/home/nop-90/Documents/sources/pwndbg/pwndbg/abi.py", line 138, in update
    abi = gdb.execute('show osabi', to_string=True).split('\n')[0]
gdb.error: Remote 'g' packet reply is too long (expected 168 bytes, got 328 bytes): 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f04ad97e00000000e047fa76100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

I've added debug flags on gdbserver and located the part where it failed. I don't know what the libthread_db.so does but it seems to be important and probably not linked to pwndbg. I use a statically linked gdbserver on my firmware but it's dependent on the libthread of the platform and the gdb i'm using is probably too recent compared to the firmware i'm debugging

stnevans commented 5 years ago

@nop-90 I agree that the root problem is probably not linked to pwndbg. I'm not totally sure that libthread_db.so is the root of the problem. I don't really see how an undefined symbol or libthread_db.so not opening would lead to the error you are getting, but I suppose it's possible.

If it's just the line abi = gdb.execute('show osabi', to_string=True).split('\n')[0] that causes an error, you could replace it with abi = 'The current OS ABI is "auto" (currently "GNU/Linux").' My guess is that you would just get the same error on a different gdb command, but it might be worth trying.

Can you confirm that the gdbserver log ends where you cut it? (At putpkt ("$OK#9a"); [noack mode]).

nop-90 commented 5 years ago

@stnevans I'm not sure myself about this but it was the only sign of an error that I saw. I looked elsewhere and this error appears quite a lot, so it's probably not linked to that particular error.

By replacing abi with a fixed string, it works. Thanks, this will at least make my life easier when putting breakpoints inside threads.

No I just cut where it was crashing there is a lot more after (line 4690 and after) : https://del.dog/xozoxuropo

stnevans commented 5 years ago

I guess we should close this issue? From what I see, it doesn't seem like something we can really fix from pwndbg. The fact that executing show osabi causes issues seems more like a gdb problem and not so much a pwndbg problem.

If anyone has further insight on this issue, feel free to reopen it.

cc-sir commented 5 years ago

I have the same problem, but the gdb is wroking normally if I don't use the pwndbg. The strangest thing is that I can target remote :1234, but if I input 'n' the problem will appear: pwndbg> show architecture The target architecture is assumed to be i386:x86-64:intel pwndbg> n Remote 'g' packet reply is too long (expected 308 bytes, got 536 bytes): 000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000803e6082ffffffff783e6082ffffffff0400000000000000c09b6582ffffffff60be034088a4ffff48b30200b48dffff000000000000000000a96c82ffffffffa0106d82ffffffff000000000000000047eafd81ffffffff4602000010000000180000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f030000000000000000000000000000000000000000000000000000000000000004000000000000f004c8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ffffffff000000ffffffffff72000810182028303840094800000000ffffffffffffffffffffffffffffffff04000000040000000400000004000000e0738d0000000000e0738d000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000 pwndbg> show architecture The target architecture is assumed to be i386:x86-64:intel what should I do?

stnevans commented 5 years ago

@cc-sir is the binary you are working on public? If not, is it indeed x86-64? Any chance you could turn on some debugging as show above in this thread? (--debug ----remote-debug, set debug remote 1).