search

Stewori / JyNI

Enables Jython to load native CPython extensions.
https://jyni.12hp.de
Other
150 stars 17 forks source link
cpython-extensions ctypes java jython native-extensions numpy python

readme

JyNI – Jython Native Interface

If not yet done, make sure to visit our project homepage at www.jyni.org.

JyNI Logo

Table of contents

  1. What is JyNI?
  2. Current state
  3. Installing JyNI
  4. Building and testing
  5. Roadmap
  6. Binary compatibility
  7. Summary of changes to Python code
  8. Copyright notice
  9. License
  10. Contact

1. What is JyNI?

A current issue of Jython hindering its wider adoption is that it does not support native extensions written for CPython like NumPy and SciPy. Since most scientific Python code fundamentally depends on exactly these native extensions, it usually cannot be run with Jython. JyNI aims to bridge this gap. It is a layer allowing Jython users to load native CPython extensions and access them from Jython the same way they would in CPython. In order to enable JyNI, you only have to put it on the Java classpath when Jython is launched. It neither requires recompiling the extension code, nor building a customized Jython fork. That means it is binary-compatible with existing extension builds.

As of this writing, JyNI does not fully implement the Python C API but it does support some important extensions, most notably the core functionalities of ctypes and NumPy.

We are constantly working to improve this support and our goal is to provide the C API needed to use the scientific stack and other extensions as soon as possible.

JyNI runs on Linux, OS-X and theoretically on various other POSIX systems (not tested). From JyNI-alpha.5 onwards, it also runs on Windows.

2. Current state

We are currently able to load a C extension into Jython, call methods and access attributes, provided that the extension uses only CPython API we have already implemented.

As of JyNI-alpha.3 JyNI performs a method for garbage collection that bridges Java's and Jython's mark-and-sweep-based approach and CPython's reference-counting-based approach as used in C extensions. JyNI's approach is explained in detail in the publication Garbage Collection in JyNI - How to bridge Mark/Sweep and Reference Counting GC., Proceedings of the 8th European Conference on Python in Science (EuroSciPy 2015), CoRR abs/1607.00825 (2016), arxiv.org/abs/1607.00825

As of JyNI-alpha.4 JyNI supports new-style classes. Support for defining new types via C API that extend existing built-in types is still limited.

As of JyNI-alpha.5 JyNI supports Windows and properly manages built-in extensions. Most notably, datetime and _winreg are now built-in. When running on Windows, JyNI enables support for the mbcs encoding in Jython.

Parse and build functions PyArg_ParseTuple(), PyArg_ParseTupleAndKeywords(), PyArg_Parse(), Py_BuildValue and related work with format strings corresponding to built-in types that are already supported. Alternatively, supported built-in types can be accessed via their C API functions.

The following built-in types are already supported:

Planned:

JyNI has been tested on

It would presumably work also on other POSIX systems. If you try it on further distributions, please consider reporting your results (see contact section).

3. Installing JyNI

This section explains how to install JyNI using prebuilt binary files. If you want to build JyNI yourself, skip to section 4. Building and testing.

Running JyNI

To run Jython with JyNI support, call (from JyNI base dir)

java -cp jython.jar:build/JyNI.jar org.python.util.jython someFile.py

On Windows use ; instead of : and \ instead of /:

java -cp jython.jar;build\JyNI.jar org.python.util.jython someFile.py

Alternatively, one can use Jython's start script (assuming you copied it to the JyNI base directory):

jython -J-cp build/JyNI.jar

To run it from a different place, you just need to adjust the paths for JyNI and Jython in the above commands.

Note that

java -cp build/JyNI.jar -jar jython.jar

is NOT suitable to run Jython with JyNI support. It does not pass the provided classpath to Jython.

To run tests and demos, see section Test Example

4. Building and testing

Test Example

DemoExtension is a CPython extension that demonstrates some basic features of JyNI. It can be built like an ordinary CPython extension as described in the Python tutorial.

Enter the folder DemoExtension and type python setup.py build. On Windows make sure that MSVC for Python 2.7 is installed. Optionally type python setup.py clean.

JyNI-Demo/src/test_JyNI.py runs tests for some basic JyNI features. You can run it via

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/test_JyNI.py

or ./JyNI_unittest.sh / JyNI_unittest.bat assuming that jython.jar is placed or (sym)linked in JyNI's directory. Otherwise you need to fix the path of jython.jar in the command.

JyNI-Demo/src/JyNIDemo.py demonstrates the use of DemoExtension from Python side. It should run perfectly with either CPython 2.7.x or Jython+JyNI. To run it, type

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIDemo.py

You can alternatively run ./JyNIDemo.sh / JyNIDemo.bat.

To see a basic demonstration that JyNI is capable of using the original datetime module (a built-in module as of JyNI-alpha.5), run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIDatetimeTest.py

To see a demonstration of exception support (on native level), run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIExceptionTest.py

To see a demonstration of Tkinter support, run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNITkinterTest.py

or ./JyNITkinterDemo.sh / JyNITkinterDemo.bat.

For a demonstration of ctypes support, run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIctypesTest.py

or ./JyNIctypesDemo.sh / JyNIctypesDemo.bat.

5. Roadmap

Further steps:

6. Binary compatibility

CPython extensions must be compiled with the flag WITH_PYMALLOC activated (which is actually the default anyway). Otherwise they must not use the macros PyObject_MALLOC, PyObject_REALLOC and PyObject_FREE directly. If they use these macros with WITH_PYMALLOC not activated, JyNI will most likely produce segmentation faults. Without the flag these macros directly map to the system's malloc function family. In that case JyNI would not be able to prepare the JyObject header in every case. However, it would be easy to recompile such extensions for JyNI. Simply add the WITH_PYMALLOC definition at compile time or replace the macros by their corresponding function calls.

In general, the less an extension hacks with CPython-specific internals, the greater the chance it will run with JyNI. Especially allocation/deallocation should not be done by hand, since JyNI must be able to setup the JyObject headers.

We hope that most extensions are not affected by this issue.

7. Summary of changes to Python code

Briefly, we took the Python files crucial for loading native C extensions and modified them to perform the explained purpose of JyNI. Mainly we changed alloc and dealloc behavior of PyObjects consequently, also when inlined like in stringobject.c or intobject.c. The new allocation behavior adds a header called JyObject before the PyObject header or even in front of PyGC_Head if present. This header contains information that allows to wrap or mirror a corresponding Jython jobject in a seamless way.

Mirror mode is used for built-in types that allow access to their data structure via macros. To keep these accesses valid, the data structure mirrors the actual Jython jobject. Syncing is not an issue for immutable objects and can be done initially for each object. One mutable built-in object needing mirror mode is PyList. We perform sync from Java by inserting a special java.util.List implementation as a backend into the corresponding Jython PyList. A bit more tricky is PyByteArray, which also features a direct access macro, namely PyByteArray_AS_STRING. However, a fix for this has low priority, as this macro is not used by NumPy. Nevertheless, we can say a bit about this issue. As PyByteArray in Jython uses a primitive byte[] array as backend, we cannot replace the backend by a subclass. Our most promising idea to solve the issue anyway is to insert a custom implementation of SequenceIndexDelegate as delegator. PyByteArray is a subclass of PySequence and thus offers a delegator field that allows to customize the indexed access behavior.

Built-ins not providing access via macros can be wrapped. The original CPython data structure is not used and not even allocated. All functions of such objects are rewritten to delegate their calls to Jython. However, also in these cases, we usually do not rewrite everything. Functions that access the data only via other functions are mostly kept unchanged.

8. Copyright notice

Copyright of Python and Jython: Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation. All rights reserved.

Copyright of JyNI: Copyright (c) 2013, 2014, 2015, 2016, 2017, 2018 Stefan Richthofer. All rights reserved.

9. License

The software in this package is distributed under the GNU Lesser General Public License.

A copy of GNU Lesser General Public License (LGPL) is included in this distribution in the files COPYING and COPYING.LESSER. If you do not have the source code, it is available at:

https://github.com/Stewori/JyNI

JyNI is partly based on source files from CPython 2.7.3, 2.7.4, 2.7.5, 2.7.6, 2.7.7, 2.7.8, 2.7.9, 2.7.10, 2.7.11, 2.7.12 and 2.7.13. As such, it includes the common license of CPython 2.7.3, 2.7.4, 2.7.5, 2.7.6, 2.7.7, 2.7.8, 2.7.9, 2.7.10, 2.7.11, 2.7.12, 2.7.13 and Jython in the file PSF-LICENSE-2. Whether a source file is directly based on CPython source code is documented at the beginning of each file.

For compliance with PSF LICENSE AGREEMENT FOR PYTHON 2, section 3, the required overview of changes done to CPython code is given in section 7 of this file.

For convenience, a copy of the current section is provided in the file LICENSE.

10. Contact

Please use the contact information provided on www.jyni.org/#contact.