croak() with full Python exception tracebacks; don't leak memory when croak()ing

[pypt]

Hi Stefan,

Would you consider pulling in the following:

• 00f007b - on Python exceptions, carp() with a full exception traceback (stack trace) instead of just a exception message. We've found it tremendously useful to have the full trace on Python exceptions, and with something like Carp::Always (or simple confess()) one can get the Perl side of the trace too.
  • Note: among many bad ways to throw a Unicode exception in Python 2, I have opted for just returning UTF-8 characters as \uXXXX as detailed in this blog post (functionality tested here).
• 3588569 - instead of allocating an exception message, passing it to carp(), never bothering to deallocate it and thus leaking memory, keep a couple of global variables with pointers to allocated data for carp() and release them on next call to croak_python_exception().
  • The use case here is that the Perl caller might be eval{}ing a call to Python code and treating the exception gracefully, so leaked exception messages (and helper objects) build up and eat through a lot of memory.

Tested and works on OS X, Ubuntu 12.04 with Python 2.7 and 3.5 (see Travis build).

Please let me know if you have any questions or need some additions.

[niner]

An interesting idea!

As an alternative implementation, what do you think about throwing an exception object instead of a string? This object could override stringification to return just the exception message itself, but also provide a method for getting the full backtrace. The stringification method could also check whether Carp::Always is active and automatically print a full backtrace in that case.

Throwing an object could also solve the memory leak, as we'd create a fully refcounted SV for the message itself.

[pypt]

Even though it's not stated explicitly in perlvar, I think everyone expects $@ to always be a string (and not some sort of an object that might be cast into a string), so returning an object instead of a string is very likely to break someone's code.

For example, Inline::Python returns objects for booleans to Perl which doesn't have a native boolean type. While it's a rather safe behavior, in our own particular case (we're using Inline::Python to gradually migrate our Perl codebase to Python) it still required some additional code changes and testing because, for example, one can't pass the returned boolean to the JSON module for serialization without enabling allow_blessed first.

So, I'd still opt for returning a vanilla string with a traceback. If someone really wants to get the exception object back, one can try ... except ... on the Python's side of the game and return the exception to Perl themselves.

[pypt]

Hi Stefan, have you maybe made a decision on this one?

[niner]

Sorry, I'm quite drained currently. Major systems restructuring, FOSDEM, quarterly management meeting. No time to think this through. Will get back to you once I've found some quiet hour.

[pypt]

Thanks Stefan, take your time!

[pypt]

Hi Stefan, still overloaded?