Request for input for JPype

[Thrameos]

I am the lead developer for JPype. As one of the fellow bridge projects I wanted to drop by to see if there are unserved needs to be addressed in the Python/Java community. Thus far there hasn't been a lot of conflicts between these two projects but I will be adding new capabilities that put them more on a collision course. SO just like the PyJNIus project, I thought it would be a good idea to get the discussion started.

JPype is currently only a forward bridge presenting Java to Python, but I will be adding a reverse bridge to allow Java to execute Python code in the near future. JPype has been very much focused on providing a high quality native Python like interface to Java classes. Over the past 2 years I have been working primarily in the backend to make the core rock solid stable, deal with leaks, and boost performance for scientific coding. I am trying to design the reverse bridge to be much the same in which Python classes have as much of a native feel in Java as possible and can be customized to allow packages like numpy, scipy and matplot lib to be used in a very native like fashion. This means the JPype solution will be doing code analysis of Python classes to probe their capabilities and then use a code generator to mixin Java interfaces to create a Java class that best represents the abilities of the class (Number like, List like, Dictionary like). This will allow Python classes to be operated on by Java using the collections libraries and such. This makes use a lot of Java ASM for code generation. I believe this is likely a different approach than JPY.

As I am working to develop the reverse bridge, I am looking at JPY to understand how other reverse bridges are presenting Java concepts. Unfortunately the documentation on this project seems pretty underwhelming so it seems like I need to recruit some JPY users to help give me a prospective on how they feel Python should be presented in Java. The other collaboration source being Jython, but as they are actually implementing rather than just providing a view it is difficult to see it as giving a native feel.

I see there has been some progress on this project in terms of commits over the past few years but I also see outstanding pull requests running back 2 years in times. How active is this project? Are there users who are in need of an alternative? Is there anyone willing to participate in the evaluation and review process for a potential alpha quality reverse bridge over the next 6 months? How do you see an ideal presentation of Python objects in Java?

[forman]

Hi @Thrameos, thanks for your feedback! Although the project is currently not very active it is still alive and jpy is still in heavy use. Our use case is the satellite data processing software SNAP, that we develop for ESA, see snap-python.

Our whole team feels very sorry about not being able to respond to PRs and issues in a timely, (cough!) ok, adequate manner. There have also been offers from skilled people to help maintaining the project. But we fear that jps`s further development would diverge from our own core requirements, which originate from efficient array data processing using Python numpy and the like using native Java array in-memory references.

I'd really like to promise improving jpy's documentation and deal with its issues in a proper way, but we are all currently occupied by other work. This may change again in the future. However, I can offer to explain some of the jpy concepts to you, and provide some core requirements that may be helpful for your implementation. Btw, we are not reluctant of using a robust, alternate bi-directional bridge one day, i.e. JPype. We developed jpy, because there was no such software at that time.

[Thrameos]

JPype also specializes in high speed transfer of buffers and arrays. As a scientist trying to provide a bridge for a group of physists that has been a major requirement. Though as a unidirectional bridge this is mainly to allow Java buffers and arrays to be used in numpy rather than the other way around. I also implemented exporting of Python buffers to Java. But write through sharing of memory is limited by the JNI. It is a shame that you cant create locked native arrays.

Perhaps you can lay out some of the requirements that jpy as sample code you would like to be able to run. I may already be close on some areas.

[Thrameos]

Perhaps I should begin by laying out JPype requirements so that we can see what is covered and what items you would add. If you read a blog post on the topic of Python/Java bridges, one could get the idea that all the projects do the same thing (JPype, JEP, JPY, PyJNIus, ...). But as they all have different implementations and different goals, (and in many cases poor documentation) it is really hard to distinguish them without getting down to a requirements/specification list. Unfortunately the other side effect of so many bridges is a divided user community and poor ability to address user needs. Hence my desire to reach out an olive branch to the other bridges to find out what about the needs that I don't understand. Some like Py4J serve a very different purpose and thus will always need to be separate, while others are just duplicative efforts which would be best to be merged.

So I will put down what I believe JPype requirements are and you can copy and paste it to mark down JPY requirements, add to the list etc. Not every JPype requirement has been realized so I will check those that we have and leave those that I see as goals unchecked.

JPype requirements/specifications are as follows. General:

• [x] Python and Java GC is linked such that both collectors operate when one exhausts memory.

Python to Java:

• [x] Python can start the JVM.
• [x] Python can create a native representation for a Java Class (JClass)
  • [x] Python can access the Java inheritance tree and probe using isinstance.
  • [x] Python can create a Java object from a class.
  • [x] Python can using reflection on a Java Class.
• [x] Python can create a native representation for a Java Array (JArray)
  • [x] Python can create multidimensional Java arrays.
  • [x] Python can convert a Java array into a Python memory buffer. (needed for numpy)
  • [x] Python can convert multidimensional Java arrays into Python memory buffers (regular vs jagged).
  • [x] Python can access Java arrays using slicing.
  • [x] Python can access multidimensional Java arrays using slicing.
  • [x] Java slices are still Java arrays.
  • [x] When a Java slice is passed to Java, the slice portion is cloned.
• [x] Python can create a native representation for Java primitives.
• [x] Python can call a Java method.
  • [x] When Python calls a Java method it always returns a Java type. (with exception of True, False, and null)
  • [x] Python can use adapters to convert Python objects to Java when calling.
  • [x] Method resolution obeys Java overloading resolution rules.
  • [x] Python can call variadic argument methods as a list of arguments.
  • [x] Python can call variadic argument methods by expanding a tuple.
  • [x] Python can call a caller sensitive method properly.
• [x] Python access a Java field.
• [x] Python must respect Java access rules (public/protected/private) for fields, methods and classes.
• [x] Python can convert a Java nio direct buffer into a Python member buffer. (needed for numpy)
• [x] Python can implement a Java interface.
  • [x] Java can call Python code implemented as a Java interface.
• [ ] Python can extend a Java class.
  • [ ] Java can call Python code that was extended from a Java class.
• [x] Python can import a Java package as a module.
  • [x] Python can dir a Java package for public members.
  • [ ] Python can add to the class path after JVM is started.
• [x] Python can customize a Java class to give it native look and feel.
• [x] Python can store a Python object in a Java structure.
• [x] Python can view Java collections as Python collections.
• [x] Python understands Java encoding for Unicode strings.
• [x] Python can register an adapter to automatically convert a Python object to Java.
• [x] Python can cast Java types (using @ operator).
• [x] Java null is represented as Python None.
• [x] Python can render Javadoc as Python docstrings.
• [x] Python can operate Java threads.
• [x] Python can synchronize on a Java object using the "with synchronized(x)" statement.
• [x] Python can terminate the JVM.
  • [x] When terminates all Java clean up is executed.
• [x] A Java thread can extend the life of the Python interpreter.
• [x] The lifetime of a Python object can be tied to the lifetime of a Java object.
• [x] When Python holds a Java object the lifetime of Java object extends to the Python object.
• [x] Python can capture a ^C without terminating when running interactively.
• [x] Python can use the "with" statement to operate on Java try/catch with resource.
• [x] Python can use the "with" statement to operate on Java closeable.
• [x] Python can catch a Java exception.
• [x] Python can throw a Java exception.
• [x] A Python exception can unwind through Java when thrown from a Java interface implementation.
• [x] Python displays a Java exception stacktrace as Python native.
• [x] Python can pass a callable or lambda to a Java functional interface.
• [x] Python can pickle a Java object.

Java to Python:

• [ ] Java can start Python.
• [ ] Java can create a native representation for a Python class.
• [ ] Java can create a Python object from a class.
• [ ] Java can call a Python method.
• [ ] Java can evaluate a Python statement.
• [ ] Java can create a Python scope.
• [ ] When Java calls a Python method it always returns a Python type.
• [ ] Java can using reflection on Python attributes.
• [ ] Java can access the Python inheritance tree and probe using instanceof.
• [ ] Java can view which Python protocols a Python object implements.
  • [ ] Java can cast a Python object to a Python protocol.
• [ ] Java can convert a Python memory buffer into Java buffer.
• [ ] Java can initiate a transfer of Python buffer into Java array.
• [ ] Java can implement a new Python class.
• [ ] Java can import a Python module as a Java package.
• [ ] Java can customize a Python class to give it native look and feel.
• [ ] Java can view Python collections as Java collections.

If you can be so kind as marking up JPY requires/specification or putting down items that you feel JPype is missing, it would go a long way to helping me define what I need to achieve in a reverse bridge.