Closed eblot closed 4 years ago
That might indeed be an issue with the implementation that I have contributed. Seems like the variable isn't in that scope.
It would be insanely helpful if you could provide a tiny example or a branch or something so I could reproduce this locally.
I'm trying to troubleshoot the gnuradio-companion application. Unfortunately, I do not know how to shrink down the issue down to a simple use case... Any chance you are using macOS/homebrew, in which case you could reproduce the whole issue?
I do not know if it can help, but enumerating frame.variables
leads to an empty content, i.e. len(frame.variables) == 0
and frame.variables['f'] == []
I forgot to report the LLDB version:
$ lldb --version
lldb-1001.0.13.3
Swift-5.0
Catching the error and returning None:
diff --git a/cpython_lldb.py b/cpython_lldb.py
index 13d6869..cef3c52 100644
--- a/cpython_lldb.py
+++ b/cpython_lldb.py
@@ -377,7 +377,12 @@ class PyFrameObject(WrappedObject):
"""
Extract PyFrameObject object from current frame w/o stack walking.
"""
- f = frame.variables['f'][0]
+ print(frame, frame.name, len(frame.variables))
+
+ try:
+ f = frame.variables['f'][0]
+ except IndexError:
+ return None
shows
(lldb) py-bt
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bb6960> >, '_PyEval_EvalFrameDefault', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae360> >, '_PyEval_EvalCodeWithName', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bb6960> >, '_PyEval_EvalFrameDefault', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae360> >, 'function_code_fastcall', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae360> >, '_PyEval_EvalFrameDefault', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae390> >, '_PyEval_EvalCodeWithName', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae360> >, '_PyEval_EvalFrameDefault', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae390> >, 'function_code_fastcall', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bb6960> >, '_PyEval_EvalFrameDefault', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae360> >, 'function_code_fastcall', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae360> >, '_PyEval_EvalFrameDefault', 0)
(<lldb.SBFrame; proxy of <Swig Object of type 'lldb::SBFrame *' at 0x106bae3c0> >, '_PyEval_EvalCodeWithName', 0)
That fix might work, but I would like to see a regression test.
Any chance you are using macOS/homebrew, in which case you could reproduce the whole issue?
yes, I do.
I do not know if it can help, but enumerating frame.variables leads to an empty content, i.e. len(frame.variables) == 0 and frame.variables['f'] == []
Seems like either the debug symbols are missing or the code was optimized out.
Hum, maybe Homebrew does not keep the debug symbols of the Python interpreter... Need to rebuild it from scratch then.
I can check later.
Rebuilding Python with --with-pydebug seems to break SWIG-based dependencies, so I guess I will need quite some time to rebuild the full stack. OMG.
Ok things are going crazy, as --with-pydebug
ends up building a Python interpreter with missing symbols such as _PyModule_Create2
, so I better follow another route.
How to obtain a Python interpreter with debug symbols that works with cpython-lldb?
So there are multiple issues with brew:
I could however get the following working and was also able to reproduce your bug:
Create segfault_python.py
with the following content:
import ctypes
def f():
c_long_p = ctypes.POINTER(ctypes.c_long)
print(ctypes.cast(1, c_long_p).contents)
if __name__ == "__main__":
f()
Then run:
PATH=/usr/bin:$PATH lldb $(which python3.7) ./segfault_python.py
Start the program with r
. You may need to hit c
because for some reason the program stopped before doing anything. Then it actually reaches the segfault. Using bt
actually confirms this. py-bt
now produces the reported bug (IndexError
).
Indeed, it looks like same issue.
I spend the last two hours trying to find a way to get a python interpreter with debug symbols with no luck... Which interpreter are you using?
I'm using Apple's lldb (which comes with xcode)
Which interpreter are you using?
Homebrew. (so $(which python3.7)
is /usr/local/bin/python3.7
). But I also don't have debug symbols and your fix just leads to an empty Python backtrace. I think this just doesn't work too well with homebrew since they don't ship debug symbols and as you have figured out, rebuilding is quite a pain.
We should however fix cpython-lldb to at least not crash in that case. I am thinking about a test vector...
(note: my piece of code was not a fix, only a way to track down which array(s) were empty.) lldb claims it cannot use system's python (because of SIP). This platform [macOS] is becoming a developer's hell...
@eblot Thank you for the bug report! This clearly identifies a gap in testing. I previously started working on adding support for TravisCI checks run on Mac OS as well, but never finished it...
I've just successfully tried this locally on my macbook with LLDB that is shipped with Mac OS:
$ /usr/bin/lldb -v
lldb-1001.0.13.3
Swift-5.0
I'm using vanilla CPython built from the source code by the means of ./configure; make
(note, that you must not add --with-pydebug
as that would produce a build that is binary incompatible with C extensions linked against a normal release build of CPython and lead to crashes).
Ok. I'd like to debug a PyGTK issue but as I'm unable to rebuild one of the many dependencies (pango/cairo) for some reason, it looks like I'm in a deadlock. I do not understand why Homebrew's python gets no debug symbol. I am definitely missing something about macOS way of handling debug information.
Thanks for the --with-pydebug
tip, at least I will not have to fight against this one :-)
I do not understand why Homebrew's python gets no debug symbol.
Yeah, I had the same issue with CPython provided via Homebrew. I'm not entirely sure what causes this problem either. The formula definition looks correct to me: https://github.com/Homebrew/homebrew-core/blob/master/Formula/python.rb (I tried passing the same flags to configure
and it worked fine for me locally)
I wonder if debug symbols implicitly get strip
ed in themake install
step.
I tweaked the formula to stop the build before the installation step, i.e.
system "make"
system "false"
system "make install"
and ran it as brew install -d -v -s python
so that it stops before the installation step, and enter a shell, from which I could dig in. At this point, I was not even able to get the debug information - or I failed to run the proper commands to get them...
I'm now trying to rebuild Python outside Homebrew to understand what's going on...
Some news about this debug symbol issue (using segfault_python.py
):
--enable-framework
, debug symbols are found and cpython-lldb has no trouble locating the backtrace information.--enable-framework
, debug symbols cannot be located, cpython-lldb is unable to show the backtrace information.It seems to boil down to this framework option, maybe there are some extra steps to enable debug symbols, or to load them from LLDB. Debug symbol management on macOS is still a foggy area for me.
cc -o Python.framework/Versions/3.7/Python -dynamiclib \
-all_load libpython3.7m.a -Wl,-single_module \
-install_name .../framework/Python.framework/Versions/3.7/Python \
-compatibility_version 3.7 -current_version 3.7 \
-framework CoreFoundation -ldl -framework CoreFoundation
libpython3.7m.a (its .o files) contain the debug information, but the resulting framework does not. There's definitely an extra step to add somewhere there to preserve or extract the debug symbols...
Ok, I think I got it. I hate tools that do black magic behind users' back, and that's exactly what Apple does.
Once the make stage is over, dSYM file can be generated for the freshly created framework with
[xcrun] dsymutil Python.framework/Versions/3.7/Python
The debug information is created as ./Python.framework/Versions/3.7/Python.dSYM
and lldb
should now be able to automatically locate and use it. In turn, cpython-lldb
is able to recover the frame information and output the expected information.
I still need to check how to integrate this step with brew formula, as it seems the make install
stage seems to ignore any dSYM directory if it exists.
Wow... I would have never suspected Homebrew to quietly trash the command line so badly!
It not only rewrites all the warnings settings - removing most of them -, but also replace -g -O3
with -Os
(say goodbye to the debug symbols...) and forces nehalem
as the CPU core.
No wonder how it is so difficult to obtain the debug symbols. Maybe it is documented somewhere, or maybe not.
-Os
is defined as "-Os is the default Apple uses for all its stuff so let's trust them", why not, it can improve cache management and the like-g
stripping. This sound like an authoritative settingAnyway, I finally got it working:
$ lldb /usr/local/opt/python3/bin/python3 -- ~/.lldb/seg.py
(lldb) target create "/usr/local/opt/python3/bin/python3"
Current executable set to '/usr/local/opt/python3/bin/python3' (x86_64).
(lldb) settings set -- target.run-args "/Users/eblot/.lldb/seg.py"
(lldb) r
Process 22588 launched: '/usr/local/opt/python3/bin/python3' (x86_64)
Process 22588 stopped
* thread #2, stop reason = exec
frame #0: 0x0000000100005000 dyld`_dyld_start
dyld`_dyld_start:
-> 0x100005000 <+0>: popq %rdi
0x100005001 <+1>: pushq $0x0
0x100005003 <+3>: movq %rsp, %rbp
0x100005006 <+6>: andq $-0x10, %rsp
Target 0: (Python) stopped.
(lldb) c
Process 22588 resuming
Process 22588 stopped
* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x1)
frame #0: 0x000000010174b4d7 _ctypes.cpython-37m-darwin.so`l_get + 4
_ctypes.cpython-37m-darwin.so`l_get:
-> 0x10174b4d7 <+4>: movq (%rdi), %rdi
0x10174b4da <+7>: movq %rsi, %rdx
0x10174b4dd <+10>: sarq $0x10, %rdx
0x10174b4e1 <+14>: je 0x10174b4fb ; <+40>
Target 0: (Python) stopped.
(lldb) bt
* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x1)
* frame #0: 0x000000010174b4d7 _ctypes.cpython-37m-darwin.so`l_get + 4
frame #1: 0x000000010174681d _ctypes.cpython-37m-darwin.so`Simple_repr + 74
frame #2: 0x0000000100114baf Python`PyObject_Str(v='0x10158a440') at object.c:562:11 [opt]
frame #3: 0x00000001000f65f9 Python`PyFile_WriteObject(v='0x10158a440', f=<unavailable>, flags=1) at fileobject.c:129:17 [opt]
frame #4: 0x0000000100173bcd Python`builtin_print(self=<unavailable>, args='0x1015c33c8', nargs=1, kwnames=<unavailable>) at bltinmodule.c:1905:15 [opt]
frame #5: 0x00000001000e947e Python`_PyMethodDef_RawFastCallKeywords(method=0x000000010027c530, self='0x101237d10', args='0x7ffeefbfe800', nargs=1, kwnames=<unavailable>) at call.c:660:18 [opt]
frame #6: 0x00000001000e8ae2 Python`_PyCFunction_FastCallKeywords(func='0x101243460', args=<unavailable>, nargs=<unavailable>, kwnames=<unavailable>) at call.c:734:14 [opt]
frame #7: 0x000000010017de82 Python`call_function(pp_stack='0x7ffeefbfe8b8', oparg=<unavailable>, kwnames=<unavailable>) at ceval.c:4568:9 [opt]
frame #8: 0x0000000100176d0b Python`_PyEval_EvalFrameDefault(f=0x00000001015c3250, throwflag=<unavailable>) at ceval.c:3124:19 [opt]
frame #9: 0x00000001000e8ebc Python`function_code_fastcall(co=<unavailable>, args=<unavailable>, nargs=0, globals=<unavailable>) at call.c:283:14 [opt]
frame #10: 0x000000010017def7 Python`call_function(pp_stack='0x7ffeefbfeac8', oparg=<unavailable>, kwnames='0x7ffeefbfea80') at ceval.c:4616:17 [opt]
frame #11: 0x0000000100176d0b Python`_PyEval_EvalFrameDefault(f=0x00000001012f0450, throwflag=<unavailable>) at ceval.c:3124:19 [opt]
frame #12: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName [inlined] PyEval_EvalFrameEx(f=<unavailable>, throwflag=0) at ceval.c:547:12 [opt]
frame #13: 0x000000010017e790 Python`_PyEval_EvalCodeWithName(_co=<unavailable>, globals=<unavailable>, locals=<unavailable>, args=<unavailable>, argcount=0, kwnames='0x7ffeefbfec78', kwargs='0x7ffeefbfece0', kwcount=0, kwstep=2, defs='0x7ffeefbfecf8', defcount=0, kwdefs='0x7ffeefbfed08', closure='0x7ffeefbfed10', name='0x7ffeefbfed18', qualname='0x7ffeefbfed20') at ceval.c:3930 [opt]
frame #14: 0x00000001001752b8 Python`PyEval_EvalCode [inlined] PyEval_EvalCodeEx(_co=<unavailable>, globals=<unavailable>, locals=<unavailable>, args=<unavailable>, argcount=0, kws=<unavailable>, kwcount=0, defs=<unavailable>, defcount=0, kwdefs=<unavailable>, closure=<unavailable>) at ceval.c:3959:12 [opt]
frame #15: 0x000000010017528d Python`PyEval_EvalCode(co=<unavailable>, globals=<unavailable>, locals=<unavailable>) at ceval.c:524 [opt]
frame #16: 0x00000001001a394b Python`run_mod(mod=<unavailable>, filename=<unavailable>, globals={u'f': '0x100632d38', u'__builtins__': '0x100632cd8', u'__annotations__': {}, u'__file__': u'/Users/eblot/.lldb/seg.py', u'__loader__': '0x100632c90', u'ctypes': '0x100632d20', u'__doc__': None, u'__cached__': None, u'__name__': u'__main__', u'__package__': None, u'__spec__': None}, locals={u'f': '0x100632d38', u'__builtins__': '0x100632cd8', u'__annotations__': {}, u'__file__': u'/Users/eblot/.lldb/seg.py', u'__loader__': '0x100632c90', u'ctypes': '0x100632d20', u'__doc__': None, u'__cached__': None, u'__name__': u'__main__', u'__package__': None, u'__spec__': None}, flags=<unavailable>, arena=<unavailable>) at pythonrun.c:1035:9 [opt]
frame #17: 0x00000001001a2975 Python`PyRun_FileExFlags(fp=0x00007fff9454d030, filename_str=<unavailable>, start=<unavailable>, globals={u'f': '0x100632d38', u'__builtins__': '0x100632cd8', u'__annotations__': {}, u'__file__': u'/Users/eblot/.lldb/seg.py', u'__loader__': '0x100632c90', u'ctypes': '0x100632d20', u'__doc__': None, u'__cached__': None, u'__name__': u'__main__', u'__package__': None, u'__spec__': None}, locals={u'f': '0x100632d38', u'__builtins__': '0x100632cd8', u'__annotations__': {}, u'__file__': u'/Users/eblot/.lldb/seg.py', u'__loader__': '0x100632c90', u'ctypes': '0x100632d20', u'__doc__': None, u'__cached__': None, u'__name__': u'__main__', u'__package__': None, u'__spec__': None}, closeit=1, flags=0x00007ffeefbfef58) at pythonrun.c:988:11 [opt]
frame #18: 0x00000001001a201b Python`PyRun_SimpleFileExFlags(fp=<unavailable>, filename="/Users/eblot/.lldb/seg.py", closeit=1, flags=<unavailable>) at pythonrun.c:429:13 [opt]
frame #19: 0x00000001001ba89e Python`pymain_main [inlined] pymain_run_file(fp=0x00007fff9454d030, filename=<unavailable>, p_cf=0x0000000000000000) at main.c:427:11 [opt]
frame #20: 0x00000001001ba820 Python`pymain_main [inlined] pymain_run_filename(pymain=<unavailable>, cf=<unavailable>) at main.c:1606 [opt]
frame #21: 0x00000001001ba80d Python`pymain_main [inlined] pymain_run_python(pymain=<unavailable>) at main.c:2867 [opt]
frame #22: 0x00000001001ba7e2 Python`pymain_main(pymain=<unavailable>) at main.c:3028 [opt]
frame #23: 0x00000001001baf80 Python`_Py_UnixMain(argc=<unavailable>, argv=<unavailable>) at main.c:3063:12 [opt]
frame #24: 0x00007fff5dcb63d5 libdyld.dylib`start + 1
(lldb) py-bt
Traceback (most recent call last):
File "/Users/eblot/.lldb/seg.py", line 9, in <module>
f()
File "/Users/eblot/.lldb/seg.py", line 5, in f
print(ctypes.cast(1, c_long_p).contents)
For the record:
I'm trying to use cpython-lldb for the first time: a PyGtk application calls CoreGraphics and crashes.
Environment: macOS 10.14.6, Python 3.7
Unfortunately,
py-bt
fails to show the backtrace, with the following error:Maybe I do not call
py-bt
how I should. Any hints would be appreciated.The native callstack is:
I tried to select a frame from the Python interpreter, but it does not help.