search

pace-neutrons / Pace-Project-Plan

0 stars 0 forks source link

Brille (Python) interface for pyHorace #138

Closed mducle closed 3 years ago

mducle commented 4 years ago

Whilst dealing with #134 we found that because Brille uses the Matlab-Python interface (Brille is primarily a Python module), it does not work with the Horace Python module which uses a compiled Matlab library so that users do not need a Matlab license.

The way that pyHorace works is that the user first launches a Python interpreter. When the user import pyHorace the code loads the compiled Matlab library which entails running a Matlab interpreter within the original Python interpreter process (called "in-process" loading).

In contrast, the way that Brille (or any other Python module) works in normal Horace use, is that the user first launches Matlab, and then loads Horace with the horace on or horace_init command; the user then loads the brillem module and internally Matlab will then load a Python interpreter within its own process (also "in-process" loading).

What happens when pyHorace tries to load brillem is that first a Matlab interpreter is loaded within the outer Python interpreter; and then the Matlab tries to load another Python interpreter within the same process and this will crash the whole process.

There are at least two solutions for this:

  1. Rather than using "in-process" loading, both Matlab and Python could use "out-of-process" loading - that is the Matlab and Python interpreters would be in separate processes and use some external communication mechanism rather than sharing memory (pipes, sockets etc). Whilst Matlab has built-in support for the "out-of-process" loading, Python does not have this so code will have to be written and the pyHorace architecture would have to be changed to handle this. It would also entail some performance losses as the processes would not be able to share memory.

  2. Use a different mechanism for Matlab to interface with Brille than the standard Matlab-Python bridge. This may have to be custom code for each Python module which we would want to support for use within pyHorace. (For normal Horace users the standard Python-Matlab interface could still be used.)

g5t commented 4 years ago

There may be a third option to investigate. brille is a C++ library with Python bindings. MATLAB can interface with C/C++ through its mex interface or by calling a C++ library directly. Setting up a MATLAB-brille interface that bypasses Python should be possible via either the mex or library interface, though it would take a bit of work.

Is pyHorace able to make use of the core Horace mex routines?

mducle commented 4 years ago

Yeah, for Brille having Matlab call the C++ directly would probably be the best option. However, there is some code in the Python wrappings (mostly constructors and the stuff in the c-to-python file) which might be useful for Matlab too.

If we're targeting the latest versions of Matlab, I would rather not write a mex file, but to use the direct C++ interface... In which case some of the code in the Python wrapping could be moved to a new Brille library dll/so target...?

mducle commented 4 years ago

Yes, pyHorace can use the core Horace mex functions.

g5t commented 4 years ago

If we're targeting the latest versions of Matlab, I would rather not write a mex file, but to use the direct C++ interface... In which case some of the code in the Python wrapping could be moved to a new Brille library dll/so target...?

Adding a new CMake target for a brille library (libbrille?) is no problem. It's only missing at the moment because most of the C++ code is not encapsulated in a namespace. I think it would be great if some of the utility functions, e.g., for initialising brille objects from Python, can be reused. I'll also need to look into creating the .mlx definition file for MATLAB.

mducle commented 3 years ago

Namespace encapsulation was completed for Brille but constructing a Matlab C++ interface using the built-in Matlab mechanism was too complicated - Brille is too large a project for the automatically generated wrapper to work - it would require extensive manual annotation to the mlx definition file. It was decided to abandon this and to keep Brille a Python application.

Instead the pyHorace bridge is done using the mex interface used for other Python calling. In this method, an API for Matlab to call Python is defined and there are two implementations. One implementation targets the use case of a Matlab user wanting to use the PACE python components (Euphonic and Brille) and the other implementation targets Python users who want to use the combination of Horace+Euphonic/Brille. The first implementation is the light_python_wrapper project and the second is the pyclasswrapper class in pyHorace, implement in these two commits.