Closed yanghao closed 2 years ago
That doesn't look like a Python exception coming from Python code called from C++ code, but more like a segfault in said C++ code. You can try enabling the debug flag: export EXTRA_CLING_ARGS=-g
, but debug symbols (and thus traceback) don't play nice with the JIT yet (WIP). The upshot being that resolving symbols from JITted code isn't possible, so the traceback doesn't show the actual C++ function called.
Now, if there were a Python exception it should trace through the C++ code back into Python just fine. Example:
import cppyy
cppyy.cppdef("""
void callit(void (*f)()) {
std::cerr << "in C++" << std::endl;
f();
}""")
def raises():
print("in Python")
raise RuntimeError("failed")
cppyy.gbl.callit(raises)
which for me results in:
in C++
in Python
Traceback (most recent call last):
File "pyexc.py", line 13, in <module>
cppyy.gbl.callit(raises)
RuntimeError: void ::callit(void(*)() f) =>
RuntimeError: failed
Thanks for the quick response.
The C++ code used in my case is pre-compiled, so I guess that's the reason why the extra flag of "-g" does not make a difference in my case. I am sure the exception is not in C++ code but in python is that I intentionally injected a "raise Exception("...")" statement. So the calling flow is something like this:
Let me try to work out a small enough example that can produce similar things and share it with you.
@wlav I tried to replicate the flow in a simple program but failed to reproduce the issue, still getting nice readable trackbacks all the time. I also did a few experiments and it looks like whenever you have an exception in python, the next line of the C code is not reached. All I did to reproduce the issue in my full-blown program is to add a simple raise Exception("") in the python callback code. However I do see something like this preceeding the non-readable-traceback:
*** Break *** abort
So I converted my program to a C++ program and used a try-catch to handle all exceptions in C++ and voila, can reproduce the problem.
So now the question is: if I got some C++ library as a black-box and I cannot change it, would there be a better way to debug it ? Is there any way for not letting C++ catch python exceptions, so cppyy can handle it properly? If those are not possible, what can be done in the C++ exception handling code that could let the python exception to propogate through back to python/cppyy?
Attaching the code below.
test.cpp:
#include <cstdio>
int (*gf)(int, int);
void add(int (*func)(int, int))
{
gf = func;
}
int f(void)
{
int r;
try {
r = gf(5, 8);
} catch (...) {
printf("can still be reached!\n");
int x = *((int *)0);
}
return r;
}
test.h
int add(int (*func)(int, int));
int f(void);
Makefile:
all:
g++ -shared -fPIC -o test.so test.c
strip --strip-all test.so
run:
python3 test.py
test.py:
from cppyy import gbl as cpp
from cppyy import cppdef, load_library, include
def func(a, b):
raise Exception("wrong ...")
return a + b
load_library("./test.so")
include("test.h")
cpp.add(func)
r = cpp.f()
print("result:", r)
I'm not 100% sure I'm following you ... If you add in the above -g
to command g++ -shared -fPIC -o test.so test.c
, don't you get the C++ function:
#9 0x00007fca404f11de in f() () from /home/wlav/cppyy-dev/test.so
so that's working as it should?
Is there any way for not letting C++ catch python exceptions, so cppyy can handle it properly?
Just to be sure we're talking about the same thing: Python exceptions are a thread-local state, not a rollback mechanism like C++ exceptions, hence there is no such thing as C++ handling Python exceptions. What happens is that on exceptional return from the callback, CPyCppyy will raise a "magic" C++ exception to do the necessary rollbacks to reach the outermost layer before returning to Python, then report the Python exception.
From the description I get so far, it is then during that rollback that there is a crash? E.g. the destructor of an object on the stack? Note that if there is a segfault because of that reason, technically all bets are off: the program is now in a corrupted state and maybe something will work, maybe not.
Best at that point, and solely for debugging purposes, is to jump out from the segfault handler back to the outermost layer. This then does not involve any exceptions (it's a longjmp
). You can enable that with:
import cppyy.ll
cppyy.ll.set_signals_as_exception(True)
See: https://cppyy.readthedocs.io/en/latest/debugging.html .
However, that will replace the original Python exception, so the traceback only points to the place where the call started in Python (here: cpp.f()
).
I've change the code in repo to now first check for Python exceptions outstanding and print them, if any, before setting the exception reporting the segfault. So now the report looks like this (with set_signals_as_exception(True)
added):
can still be reached!
*** Break *** segmentation violation
...
#9 0x00007f0787a7f1de in f() () from /home/wlav/cppyy-dev/test.so
...
#31 0x00000000005fa5ce in _start ()
Python exception outstanding during C++ longjmp:
Traceback (most recent call last):
File "test.py", line 9, in func
raise Exception("wrong ...")
Exception: wrong ...
Traceback (most recent call last):
File "test.py", line 16, in <module>
r = cpp.f()
cppyy.ll.SegmentationViolation: int ::f() =>
SegmentationViolation: segfault in C++; program state was reset
Note that checking for a Python exception and printing it is not without pitfalls after a segfault.
With that, for me anyway, there is now the traceback through Python onto the point where C++ was entered, the C++ trace including the library function where the problem occurred, the fact that it was a segfault, and the last exception seen from Python before handling the segfault.
Aside, it is actually safe to always run with set_signals_as_exception(True)
, just that it incurs a performance penalty, which is why it's not enabled by default.
Just to be sure we're talking about the same thing: Python exceptions are a thread-local state, not a rollback mechanism like C++ exceptions, hence there is no such thing as C++ handling Python exceptions. What happens is that on exceptional return from the callback, CPyCppyy will raise a "magic" C++ exception to do the necessary rollbacks to reach the outermost layer before returning to Python, then report the Python exception.
You are right, I do not really understands how C++ seemingly captured Python exceptions, but that for me is more a symptom rather than understanding how it actually works. But still, cppyy raises an C++ exception, and C++ captured that C++ exception. Thanks for clarifying on this.
I've change the code in repo to now first check for Python exceptions outstanding and print them, if any, before setting the exception reporting the segfault. So now the report looks like this (with
set_signals_as_exception(True)
added):can still be reached! *** Break *** segmentation violation ... #9 0x00007f0787a7f1de in f() () from /home/wlav/cppyy-dev/test.so ... #31 0x00000000005fa5ce in _start () Python exception outstanding during C++ longjmp: Traceback (most recent call last): File "test.py", line 9, in func raise Exception("wrong ...") Exception: wrong ... Traceback (most recent call last): File "test.py", line 16, in <module> r = cpp.f() cppyy.ll.SegmentationViolation: int ::f() => SegmentationViolation: segfault in C++; program state was reset
This is really wonderful! Thank you! I am looking forward to see it in the next release, maybe in 2.3.2? :-)
Aside, it is actually safe to always run with
set_signals_as_exception(True)
, just that it incurs a performance penalty, which is why it's not enabled by default.
Noticed and thanks!
Yes, it'll appear in 2.3.2, but you can also build from source, see: https://cppyy.readthedocs.io/en/latest/repositories.html#building-from-source-1 where you only need to checkout CPyCppyy
and build/install (see at the bottom of the linked text).
@wlav , thanks. Which change/commit was it that enabled the python traceback prints? Also want to learn a bit of cppyy internals.
Nevermind, found it. :-)
Which change/commit was it that enabled the python traceback prints?
It's not a traceback: that one is always at the end. The extra printout is the outstanding exception when entering back into C++ the first time. It's probably possible to collect a traceback of the Python calls through several layers of C++ and Python, but not (currently) the C++ calls as these originate from JITed wrapper calls w/o debugging information (the wrapper calls themselves could be tracked and reported, but would be incomplete).
In any case, the commit is here: https://github.com/wlav/CPyCppyy/commit/ba831422388afde06d6a5fb33a7a5b865a5f050c
Thanks for the explanation. the outstanding exception information is helping a lot to work with buggy shared libraries :-)
Recently realized that when calling python code from c++ code, and if there is an exception in python code, the resulting callback trace is difficult to read, something looks like this below. is there any better way to handle those kinds of situations?
cppyy version: 2.3.1 os: ubuntu 20.04