ninia / jep

Embed Python in Java
Other
1.33k stars 150 forks source link

Pass JepException back into jep #319

Open jsnps opened 3 years ago

jsnps commented 3 years ago

Hey,

Is it possible to pass some caught JepException back into python to handle the original python exception in python code? I've read that Jep is able to throw back some JepException into python, I guess this is the case for reentrant calls (e.g. java -> python -> java -> python). Background is, that I want to print the python stack trace when runnning a script and something goes wrong. So something like this:

        try (Jep jep = new SharedInterpreter()) {
            try {
                jep.runScript("faultyScript.py");
            } catch (JepException e) {
                //how to get the exception back into python?
                jep.set("ex", e);
                jep.exec("import traceback");
                jep.exec("traceback.print_exception(ex, ex.args, ex.__traceback__)");
            }
        }

The issue here is that the exception is a PyObject, a simple python wrapper around the java exception object.

So far I only see some alternative like this:

        try (Jep jep = new SharedInterpreter()) {
            try {
                jep.eval("try:\n" +
                    "  exec(open(\"faultyScript.py\").read())\n" + 
                    "except Exception as e:\n" + 
                    "  import traceback\n" + 
                    "  traceback.print_exc()\n" + 
                    "  raise e\n");
            } catch (JepException e) {
                //more handling on java side
            }
        }

Although, for this approach I would need to massage the stacktrace a bit. Any other ideas?

sys.last_value only seems to be set in an interactive interpreter, sys.exc_info() also doesn't seem to be set, which is kind of excepted, because the exception is caught on the java side.

Second alternative, add a showException flag to runScript and do this in C?

bsteffensmeier commented 3 years ago

There is currently no way to get the python exception from a JepException. I have considered adding the python exception to the JepException as a PyObject but it never seemed like that would be particularly useful and it could cause some confusion for cases where the JepException is thrown outside the scope of the Interpreter which created the PyObject. If you have a need for that info though it is something I would consider merging if you can make a pull request.

You may be able to create your own java Exception class that contains a PyObject. I don't think you could throw that exception directly form python but you could have a java function throw it. Your exception containing the PyObject would then be the cause of the JepException. See the code below for an example. The traceback wasn't working but the idea of storing the exception as a PyObject seems workable. You may need a separate field in the PythonExceptionWrapper for the traceback to make sure it gets across. You could store any state information you want in the PythonExceptionWrapper for use later in python.

class Test {

    public static class PythonExceptionWrapper extends RuntimeException {
        public final PyObject cause;

        public PythonExceptionWrapper(PyObject cause) {
            this.cause = cause;
        }
    }

    public static void throwPython(PyObject object) {
        throw new PythonExceptionWrapper(object);
    }

    public static void main(String[] args) {
        try (Jep jep = new SharedInterpreter()) {
            try {
                jep.set("Test", Test.class);
                jep.eval("try:\n" +
                         "  exec(open(\"faultyScript.py\").read())\n" +
                         "except Exception as e:\n" +
                         "  Test.throwPython(e)");
            } catch (JepException e) {
                PyObject pyExc = ((PythonExceptionWrapper) e.getCause()).cause;
                jep.set("ex", pyExc);
                jep.eval("import traceback");
                jep.eval("traceback.print_exception(ex, ex.args, ex.__traceback__)");
            }
        }
    }
}
jsnps commented 3 years ago

Ok, I see. Isn't it also that the stored PyObject would become invalid / unusable as soon as it leaves the interpreter scope (i.e. the exception leaves the try with resource block)? In this case, I agree that it would be a cumbersome feature. I will think about it again and check if it is really needed.

With regards to exceptions I have another two small questions and I don't want to flood you with too many new issues ;)

  1. For me it would be useful to throw typed python exceptions from the java side. This is already possible, but not very handy. Maybe there is an easier way, but if not, what do you think about some enum or constants for making the translation between java and python easier? I even needed to extend JepException to be able to access / define the type. The translation though works as expected then. :) To simplify this a bit, I could once fetch all longs (types) from one interpreter and hope that these are constant also among multiple jep instances (on differen threads).
    public static class TypedJepException extends JepException {

        private static final long serialVersionUID = -4282626424199367439L;

        public TypedJepException(String message, long type) {
            super(message, type);
        }
    }

    public static void throwPythonException(String message, long type) throws JepException {
        throw new TypedJepException(message, type);
    }

    @Test
    public void testThrowFromJava() throws JepException {
        try (Jep jep = new SharedInterpreter()) {
            try {
                jep.set("test", RawJepInterpreterTest.class);
                jep.eval("test.throwPythonException('some name error', id(NameError))");
            } catch (JepException e) {
                assertEquals("<class 'NameError'>: some name error", e.getMessage());
            }
        }
    }

Also imaginable could be some factory methods (this is how jython did it):

    public static PyException TypeError(String msg) { return new PyException(Py.TypeError, msg); }
    public static PyException AttributeError(String msg) { return new PyException(Py.AttributeError, msg); }
    public static PyException AssertionError(String msg) { return new PyException(Py.AssertionError, msg); }
    public static PyException IndexError(String msg) { return new PyException(Py.IndexError, msg); }
    public static PyException RuntimeError(String msg) { return new PyException(Py.RuntimeError, msg); }
    public static PyException ValueError(String msg) { return new PyException(Py.ValueError, msg); }
[...]

//and then

throw Py.ValueError("Sorry this is an invalid value :(")

Same for checking python exception on the java side - e.g.:

jepException.matches(PyExceptionType.ValueError)
  1. We are running Jep inside an Eclipse application and we need to have control over the shutdown, so we need to be able to intercept SystemExit exceptions. I digged a bit into PyErr_PrintEx but I didn't find a way to generally avoid process exits from inside the embedded python (see also here). Maybe _Py_GetConfig()->inspect could be set, but I don't know what other effects this would have. For now I just disabled the check in jep and it seems to work, I receive the system exit exception in java. Do you have any concerns about this? I guess python will get the chance to properly cleanup also on a java exit anyways. What do you think about making this behavior configurable in jep?
bsteffensmeier commented 3 years ago
  1. I agree we need some way to throw and ideally also catch specific Python exceptions from Java. One suggestion I have seen before is that we could have subtypes of jepExeption for all the standard python exception types. This would be very convenient for those types but I have generally been opposed to this because it doesn't provide any mechanism for exception types created by python libraries or user defined exception types. In general I have wanted to add the ability to ahve user supplied default conversion for covnerting objects between java and python, if we had such a framework it could be applied during exception handling to let developers define their own conversion, but this is a very large task that is unlikely in the near feature. I think it would not be too difficult to change the long pointer to a PyObject in JepException and create a constructor that allows the exception type. Although for most exception types the pointer will not change it would be safer and more flexible to use a PyObject instead.
  2. Now that you pointed it out I agree I don't like the idea of exiting an entire process because of a python call. If embedded python is a small part of your application then it is nuts to let it shut the whole thing down. The only time I would really expect jep to actually exit is in something like the jep command line program where the entire process is supposed to emulate python. Even in that case I would think it would be better to throw some sort of SystemExit exception back to java and let java do the actual exit, however that goes back to item 1 where we don't map exception types very thoroughly. Until we can solve all of our other exception issues I am in favor of a boolean in PyConfig or jepConfig for disabling the exit there.
jsnps commented 3 years ago
  1. I ended up doing something like this:

    /* package */ static synchronized void initializeExceptionTypes(Jep jep) throws JepException {
        if (exceptionTypes == null) {
            Set<PyExceptionType> ignore = Sets.newHashSet();
            ignore.add(PyExceptionType.StandardError);
            Map<Long, PyExceptionType> types = Maps.newHashMap();
            for (PyExceptionType type : PyExceptionType.values()) {
                if (ignore.contains(type)) {
                    continue;
                }
                types.put(jep.getValue(String.format("id(%s)", type.name()), Long.class), type);
            }
            exceptionTypes = types;
        }
    }

    Once initialized, it's quite convenient to map in both directions (the ignore set here is for the StandardError, which has been removed from Python 3 - still need that type for the Jython (2.7) backend) In this case the long pointer is actually better for me, because it is thread independant (and today PyObjects are not).

  2. For now I just hardcoded it in C, but once I made it configurable, I will create a pull request.