Debug Symbols on Ubuntu

[dlecocq]

Having installed python2.7-dbg, the automatic loading of the symbol file isn't working:

Pyringe (Python 2.7.3) on linux2
For a list of debugger commands, try "help()". (python's help is available as pyhelp.)
==> pid:[None] #threads:[0] current thread:[None]
>>> attach(8298)
WARNING:root:Failed to automatically load symbol file, some functionality will be unavailable until symbol file is provided.
==> pid:[8298] #threads:[0] current thread:[None]

It seems that the default SYMBOL_FILE is relative to payload, and it seems to not be present. Changing the default to, say, the binary or shared object files installed by the python2.7-dbg package then fails to pass the sanity check.

Am I missing something obvious? Does the python (in our case, a daemon) have to have been invoked with python-dbg (we were under the impression that was not the case)?

[TehMillhouse]

You shouldn't have to use the python-dbg binary, your assumptions are correct.

Unfortunaltely, I don't have a Ubuntu machine handy to debug this issue. Other people have had trouble with Ubuntu's gdb (see issue #16, though this looks different, make sure your gdb isn't compiled to use libpython3.*.so).

The fact that you're not even seeing the number of threads after telling pyringe to attach makes me think gdb just crashes. I'd appreciate if you could post basic system information (that is the version of Ubuntu, gdb, and the output of ldd /usr/bin/gdb), as well as the output of running __repl__.inferior.gdb.is_running in the pyringe repl after attempting to attach to a process. (If gdb isn't running, it will complain about None not being a valid pid.)

Gdb should search for debugging info in /usr/lib/debug/<path_to_inferior_binary> all by itself, so if you've got python2.7-dbg installed (as indeed you do), gdb should look in /usr/lib/debug/usr/bin/python2.7 all by itself (which can be verified by simply executing gdb python2.7 and looking at what gdb prints). Because of this, I'm assuming your python build is somewhere where gdb can't find the corresponding debug file by itself.

Is this a custom build? (That would explain the failing sanity check -- the symbols from python2.7-dbg likely won't work for all debugger functions) If so, getting gcc to produce debugging info in a separate file and pointing pyringe to that file using setsymbols(path) should get you somewhere.

Given more information about your system, I'll try to set up a similar system to properly debug this.

[dlecocq]

WRT gdb, it definitely links against the version of python I'd expect.

$ ldd $(which gdb) | grep python
    libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007fa9f4723000)
$ file /usr/lib/libpython2.7.so.1.0 
/usr/lib/libpython2.7.so.1.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x5d15da4466c0f66a892a5fb67187fcc2300f5c87, stripped

GDB does, in fact appear to die -- it appears as defunct, though the __repl__.inferior.gdb.is_running line says that it's running:

>>> __repl__.inferior.gdb.is_running
WARNING:root:Failed to automatically load symbol file, some functionality will be unavailable until symbol file is provided.
True
==> pid:[11575] #threads:[0] current thread:[None]

But looking for a corresponding gdb process:

$ ps aux | grep gdb
root     12755  0.2  0.0      0     0 pts/5    Z    19:58   0:00 [gdb] <defunct>

We do in fact have the /usr/lib/debug/usr/bin/python2.7, though the point about which I'm confused is this bit in pyringe. Running gdb independently, we've confirmed that we are at least referencing /usr/lib/debug/usr/bin/python2.7, but it seems like independent of what gdb loads of its own accord, the auto loading of the symbol file is problematic. At least, that's one of the places where we've seen warnings emitted.

This is not a custom build -- just the normal python2.7 package provided by a vanilla Ubuntu 12.04 machine. FWIW, we use Vagrant pretty heavily, and it's quite handy for debugging issues like this.

[TehMillhouse]

TL;DR: Ubuntu doesn't package a symbol file with type information for the normal python binary, there's nothing pyringe can do in that case. (closing this)

After setting up a basic 12.04 VM and looking at the file included in python2.7-dbg (I didn't need a VM for that, but oh well), I noticed the type information is missing (you can verify this in gdb with info types after having loaded the symbol file: if that prints out a bunch of struct definitions, you're fine, if all it gives you are __bss_start, _edata and _end, it's missing. The Build ID of my binary and debug info are identical, so this seems to be the "right" package/file). This is rather surprising and quite frustrating, as pyringe needs that information.

That's why the sanity check is failing -- it's legitimately reporting that the debug information is incomplete.

As for why gdb isn't loading the right symbol file by itself, I'm still investigating that (adding more logging and more exact warning messages in the process). As you said, auto loading of symbol files is problematic as it is right now.

As it stands, pyringe would have worked after explicitly telling it where to find the symbol file, if that file had passed the sanity check. Without type information there won't be any backtraces, file listings, proxy objects. Code injection might work if we're to allow pyringe to use non-debugging symbols, but not without some patches, and even then, you'd be mostly flying blind, as you'd have no way of knowing which thread (and under which frame) you're injecting code...

Bottom line: The python-dbg binary has type information. It's slower than the regular build and very verbose, but unless you can find a symbol file for your build of python that actually includes all the information pyringe conceptually needs (as is provided by Debian and Fedora and can be done with Gentoo, to name a few), that's the only python binary Ubuntu provides that has a chance of being used with pyringe. I'll see if I can get a hold of python2.7-dbg's maintainer and talk to him about including type information in future versions.

---

About that part you linked to in inferior.py: The idea is to allow users of pyringe-the-repl to feed their own symbol file path, allow users of pyringe-the-module to override the payload directory and symbol file locations, and to allow would-be-packagers to bundle their own symbol file if a distro doesn't distribute one as a package and they got their hands on a compatible file. (Before you say anything, yes, there are some issues with the error handling there and some misleading logging. This would have been easier to pinpoint with better logging, I'm working on it.)

[TehMillhouse]

I'll add a note about the situation on Ubuntu to the README to warn other people about this issue.

[dlecocq]

Thanks for looking into it. That's pretty unfortunate that Ubuntu doesn't package the debug symbols for the default installation. I did check out the python-dbg binary, but what ultimately turned us off to it was having to rebuild a bunch of packages for it.

Again, thanks for digging into it, and if we find an easy way to get it going on Ubuntu, we'll post it back here.

[upadhyay-prashant]

dlecocq when will you post it back ?? :o i hit the same issue on ubunbtu!

[dlecocq]

Unfortunately, I don't think we ever did find a good workaround for this :-/