Debugging python code from java callback

[ghost]

It seems that when python code is called from a java FunctionalInterface that pydevd cannot debug it. I suspect it is because pydevd cannot see the java threads. Is there anything I can do about this? The exact version of pydevd being used is what is distributed with the latest release of vscode and the python extensions. I'm using python 3.10 with the recent ssize_t patch. Note that it is the only patch applied to the source distribution from pypi.

I tried creating a python thread at the start of the python code so that it would be running in a python thread but it still didn't work. Maybe because a java thread started it?

[Thrameos]

I am not aware of any way that pydevd would be able to see Java called code. Perhaps there is some way to hook to the entry point for Java to Python calls in native/common/jp_proxy.cpp:Java_org_jpype_proxy_JPypeProxy_hostInvoke. Is there any guides on what would be required to support external calls from pydevd?

[Christopher-Chianelli]

These seem relevant: https://docs.python.org/3/c-api/init.html#profiling-and-tracing

My guess is the call look like:

JP_TRACE("Call Python");
 // new code
PyFrameObject *frame = /*somehow create frame */
Py_tracefunc(callable.get(), frame, PyTrace_CALL, Py_None);
// end of new code
JPPyObject returnValue = JPPyObject::call(PyObject_Call(callable.get(), pyargs.get(), NULL));

JP_TRACE("Handle return", Py_TYPE(returnValue.get())->tp_name);

[Christopher-Chianelli]

(well, it appears Py_tracefunc is a type that is used for tracers/debuggers, so the above probably won't work)

[Christopher-Chianelli]

As far as I am aware, PyCharm uses pydevd for debugging, and adding a breakpoint works:

package org.acme;

import java.util.function.Function;

public class MyClass {
    public static String apply(Function<String, String> mapper) {
        return mapper.apply("1");
    }
}

import jpype
from jpype import JClass
import jpype.imports

jpype.startJVM(classpath=['...'])

def my_function(arg):
    return f'Number {arg}' # Breakpoint here

MyClass = JClass('org.acme.MyClass')
assert str(MyClass.apply(my_function)) == 'Number 1'

However, if I used a separate thread, the debugger does not work:

package org.acme;

import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.function.Function;

public class MyClass {
    public static String apply(Function<String, String> mapper) throws ExecutionException, InterruptedException {
        ExecutorService executor = Executors.newSingleThreadExecutor();
        Future<String> out = executor.submit(() -> mapper.apply("1"));
        return out.get();
    }
}

import jpype
from jpype import JClass
import jpype.imports

jpype.startJVM(classpath=['...'])

def my_function(arg):
    return f'Number {arg}' # Breakpoint here

MyClass = JClass('org.acme.MyClass')
assert str(MyClass.apply(my_function)) == 'Number 1'

[Christopher-Chianelli]

So I believe this issue is specific to multi-threading, in particular when the thread that calls the python code is not the same as the thread that was the entry point for python calling java code.

[Christopher-Chianelli]

Workaround for Python (this works):

import jpype
from jpype import JClass
import pydevd
import jpype.imports

jpype.startJVM(classpath=['target/issue-reproducer-8.11.0.Final.jar'])

def my_function(arg):
    # add this code; it connect the debugger to the background thread
    pydevd.connected = True
    pydevd.settrace(suspend=False)
    # end of addition
    return f'Number {arg}'

MyClass = JClass('org.acme.MyClass')
print(str(MyClass.apply(my_function)))

(edited from https://stackoverflow.com/a/3242780 )

[ghost]

I am not aware of any way that pydevd would be able to see Java called code. Perhaps there is some way to hook to the entry point for Java to Python calls in native/common/jp_proxy.cpp:Java_org_jpype_proxy_JPypeProxy_hostInvoke. Is there any guides on what would be required to support external calls from pydevd?

Sorry for not getting back to you on this. This is a comment from the stack overflow thread mentioned below. https://stackoverflow.com/a/3114758 I'm going to try grabbing the main java thread and see if setting it there helps. I don't know the specifics yet but since settrace is standard python then, if applicable from the java side, it might be worth doing it automatically on thread creation from the main java thread. I'm really just brainstorming here and I don't know what the java api provides for this sort of thing if anything at all.

Workaround for Python (this works):

import jpype
from jpype import JClass
import pydevd
import jpype.imports

jpype.startJVM(classpath=['target/issue-reproducer-8.11.0.Final.jar'])

def my_function(arg):
    # add this code; it connect the debugger to the background thread
    pydevd.connected = True
    pydevd.settrace(suspend=False)
    # end of addition
    return f'Number {arg}'

MyClass = JClass('org.acme.MyClass')
print(str(MyClass.apply(my_function)))

(edited from https://stackoverflow.com/a/3242780 )

Thank you very much, I will give this a shot in an hour or so. Would make a good function wrapper imo.

@Christopher-Chianelli that did work thank you!

[Thrameos]

I think it is safe to label this as a won't fix as the issue is particular to the pydev and there is no way that we could make hooks do this in general without a huge performance hit. Perhaps someone would be so kind as submitting a PR to the doc so that we can move it from the issues list to the userguide?