Micro manager should be able to control micro simulations written in C++

[IshaanDesai]

Currently the Micro Manager is constructed in a way that is can only control micro simulations written in Python. The user gives a path to a micro simulation script written in Python, in the manager configuration file. However it can be the case that a user has a micro simulation script written in some other language like C++, C of even Fortran. For the time being the manager can be updated in a way that scripts written in C and C++ can be used. Both C and C++ scripts cannot be used directly, and in most cases will be built into a shared library (.so) file. The Micro Manager needs to take the shared library file and run the appropriate functions.

For micro simulations written in C++, pybind11 was the tool used in https://github.com/precice/micro-manager/pull/22.

[fsimonis]

A possibly easier (less-invasive) solution is to restructure the C++ code into a shared library, then dynamically load and call it. The user has to restructure the code anyhow.

Here is a writeup on how to do this.

[IshaanDesai]

An example of what @fsimonis is describing is:

• The C++ file: https://github.com/precice/aste/blob/1.1.0/src/metisAPI.cpp
• Loading the shared library file and using it in a Python script: https://github.com/precice/aste/blob/53a4f0caf6e43ca787c7eff148bb51c19cfe81cc/src/partition_mesh.py#L172

[fsimonis]

Quick drawing of this

[erikscheurer]

At the moment, I am looking into the shared library approach, which seems doable. Is there a way to pass python dictionaries between the languages? This is how the micromanager handles the information passing to the micro solver at the moment. They can contain different amounts of keys-value pairs and can contain either floating points or lists of floats, from what I can tell. So declaring the input to the c++ solve function as std::map<std::string, std::vector<float>> macro_data only works if we pass the scalar data also as vectors, but thats probably to much overhead?

[fsimonis]

@erikscheurer If you want to keep this as simple as possible, then transposing this could help.

A map is a list of pairs of string and lists of float [(string, [float])]. Simpler could be to split map.keys and map.values, essentially resulting in a pair of a list of strings (keys) and a list of lists of float (values) ([string], [[float]]) To flatten this even more, you can split the list of list of float into a pair of a list of lengths and a single list of all contents: ([size], [float])

A very basic interface could thus be: ([string], [size], [float]) Or in C-ish solve(int entries, string* keys, int* sizes, float*values).

This looks ugly though. Maybe pybind11 isn't that bad after all.

[erikscheurer]

The current approach would be to split up the dictionary in the python wrapper and create new c compatible variables. Meaning we would iterate through the macro_data dictionary and differentiate between floats and ndarrays to create pointers to this data. Then pass these new pointers to the c++ side with self.MicroSimulation_lib.MicroSimulation_solve(self.obj, *write_data_pointers, c_dt, *read_data_pointers) (Here the * is to split the different pointers in the list in python) This would make it possible to have different combinations of scalar and vector data. However, it would force the user to implement the c++ side in exactly the order given in the micro-manager-config.json otherwise python would change up the order of arguments passed to c++. The order of python dictionaries is also only guaranteed for Python 3.7+ What do you think of this approach?

And I agree, pybind11 looks better by the minute, but we also don't know if it would be more usable than this.

[IshaanDesai]

However, it would force the user to implement the c++ side in exactly the order given in the micro-manager-config.json

I feel that this is reasonable.

One of the main reasons why I am not convinced about the applicability of pybind11 here is because by design pybind11 is a way to write Python bindings for a C++ code. We do not want the user to write Python bindings for their micro simulation, just because we developed the Micro Manager in Python. The user should have the full freedom of working in C++ and then use the Micro Manager. But I am open to discussion here, especially if you feel that I have misunderstood how pybind11 works.

[fsimonis]

@IshaanDesai Creating an interface that is loadable and callable as a shared library from python is essentially like writing C bindings for the microsimulation (python) and the C++ code, as one needs to restructure data on both sides. This leads to type conversion challenges @erikscheurer is currently vividly exploring.

pybind11 maps python types, which simplifies the interfaces as at least the python microsimulation can directly call the code. On the other side, the code can iterate directly over a wrapper type like pybind11::dict.

flowchart LR;

A([uSim.py]) --> B(["uSim.c\n(how to call C)"]) -.-> C(["code.c\n(how to be called from C)"]) --> D([code.c++]);
A -.-> E(["code.py via pybind11\n(how to be called from python)"]) --> D;

[IshaanDesai]

Initial prototype of C++ bindings for manager API is now available through https://github.com/precice/micro-manager/pull/22

[IshaanDesai]

Resolved in https://github.com/precice/micro-manager/pull/22