How to rustify libmultiprocess ?

[ariard]

I would like to enable libmultiprocess across different languages where ProxyClient is in a C++ process and ProxyServer is in a Rust process (or vice-versa), while still keeping the interface description reconciliation safe and straightforward.

After a bit of research, I think there is 2 different approaches

• libmultiprocess.rs : rewriting libmultiprocess in rust, including a new rust-mpgen binary relying on Cap'n Proto rust compiler plugin
• rust-libmultiprocess.rs : writing Rust bindings for libmultiprocess API (src/mp/proxy.h, src/mp/util.h) and a Rust FFI generator for the artifacts of mpgen

In the case of the first approach, I think API consistency across process would be guaranteed by consuming the data definition (*.capnp). If the calculator.rs/calculator.capnp doesn't match a error should happen at Rust process compilation. This approach comes also with the benefit that if libmultiprocess features want to be used across Rust only client-server processes, the C++ code doesn't come as a dependency. That said a new dependency is added on the rust compiler plugin and it's a lot of development code and maintenance to guarantee behavior consistency and feature compatibility of the both library.

For this reason, I think the second approach is wiser.

AFAICT, you have 6 files output by mpgen:

• calculator.capnp.c++ (generated by the capnp proto compiler though called by mpgen)
• calculator.capnp.h
• calculator.proxy-client.c++
• calculator.proxy-server.c++
• calculator.proxy-types.c++
• calculator.proxy.h

I think they're mostly stub code relying on code in include/mp/proxy-types.h and as such should be okay to generater FFI interface for them. I've a doubt if there is a need to cover the calculator.capnp.c++, as I'm not sure if will be consumed by the Rust code anyway. W.r.t to libmultiprocess API, it's nice to have coverage for proxy.cpp/util.cpp and as such rebuild the process orchestration API available in Bitcoin Core's ipc/* on the Rust-side to have consistent process behavior.

Once you have *.rs output by a Rust FFI generator you can integrate them in your Rust build system. Of course, the C++ generated code must still be build by a C++ compiler, and the outcome of which linked in your Rust binary, otherwise you'll have missing symboles throw at you by the linker.

Here a graphical illustration of the second rust-libmultiprocess approach.

             ---|                   | - calculator.capnp.c++                 
        API     |                   | - calculator.capnp.h                   
                |-----> mpgen ----> | - calculator.capnp.proxy-client.c++   --------------------------> c++-compiler ----------------------------------------------------------------> cpp.o ----|
        data    |                   | - calculator.capnp.proxy-server.c++                                                                                                                        | 
             ---|                   | - calculator.capnp.proxy-types.c++                                                                                                                         |-----> linker ----> calculator.bin
                                    | - calculator.capnp.proxy.h            --------|                               | - calculator.capnp.rs                                                      |
                                                                                    |                               | - calculator.capnp.proxy-client.rs                                         |
                                                                                    |                               | - calculator.capnp.proxy-server.rs  ---------> rust-compiler --> rust.o ---|
                                                                                    |---------> rust-mpgen -------> | - calculator.capnp.proxy-types.rs                   ^
                                                                                                                    | - calculator.capnp.proxy.h.rs                       |
                                                                                                                                                                          |
                                                                                                                                                                          |
                                                                                                                                                proxy.rs -----------------|
                                                                                                                                                util.rs

Do you think the approach advocated is reasonable ? Feel free to raise all relevant points I'm missing.

This is in the context of the Altnet project where I would like to offer the choice between C++/Rust to the pluggable transports daemon writers. Starting by myself :)

[ryanofsky]

I will read this in more detail, but my first impression is there should be 0 interaction between rust and libmultiprocess. libmultiprocess is just a c++ tool that makes it easier to use capnproto from c++ code, and easier for c++ programs to make function calls across sockets using capnproto.

If you want to make a rust equivalent to libmultiprocess, I'm sure that's possible, but it would probably look a lot different just because of the different ways the build system and macros work in rust. Probably easiest just use capnproto directly from rust without writing an equivalant libmultiprocess library

[ariard]

If you want to make a rust equivalent to libmultiprocess, I'm sure that's possible, but it would probably look a lot different just because of the different ways the build system and macros work in rust. Probably easiest just use capnproto directly from rust without writing an equivalant libmultiprocess library

Yes I guess that's the first approach I'm suggesting. What I'm really interested with is bitcoin-node servicing as a ProxyServer to orchestrate Altnet processes management (start/stop, unified RPC handling by bitcoin-cli) and as a ProxyClient to let Altnet processes access validation engine/blocks/headers/etc. Though in fact it's more the src/interfaces/ which matter, ProxyServer/ProxyClient are just implementations of it.

Note, I don't have yet a clear understanding of the feature boundaries between what is a libmultiprocess mechanism and a capnproto one. I think the features I care about is interface pointers passing to conserve. bidirectional requests and object reference support. Thanks, i'm going to keep thinking what's the best fit.

[ryanofsky]

Yes I think I basically just described your first approach, because I think your first approach seems a lot simpler than the second. You can use the same altnet .capnp files from both the c++ side and the rust side, maybe with a small support library on the rust side to help deal with libmultiprocess context/threadmap arguments if your API is using them.

There should be no consistency issues in any case. The altnet .capnp files define your cross-socket interface, and the c++ and rust code both must conform in order to build.

Trying to use FFI to call c++ libmultiprocess code from rust seems like it would be painful. Trying to integrate c++ and rust build systems also would seem painful. I think if you just use https://github.com/capnproto/capnproto-rust to call capnp interfaces you need to call from rust, and use it to implement capnp interfaces you need to implement in rust, that would give you c++ <-> rust interop. The only part of libmultiprocess that you might need to reuse is the one .capnp file https://github.com/chaincodelabs/libmultiprocess/blob/master/include/mp/proxy.capnp

I'll take a look, and maybe I could help by adding a c++ <-> rust example

[ryanofsky]

Note, I don't have yet a clear understanding of the feature boundaries between what is a libmultiprocess mechanism and a capnproto one. I think the features I care about is interface pointers passing to conserve. bidirectional requests and object reference support. Thanks, i'm going to keep thinking what's the best fit.

Yes, I think almost of the features you'd be interested in are provided by capnproto, not by libmultiprocess. The main things libnultiprocess provides on top of capnproto are:

• A synchronous wrapper around capnpproto's asynchronous API, so you can call a method and have it block synchronously without having to deal with asynchronous promise objects. You may or may not want something equivalent to this in rust, but it should be easier to just write it in rust instead of trying to call the blocking c++ wrappers with FFI.
• Mapping of c++ types like std::optional, std::vector, std::map, std::tuple to capnproto types. This functionality wouldn't be useful to rust, because rust has it's own data types
• Mapping of c++ pointer/reference output arguments to capnproto's multiple return values. Mapping of c++ exceptions to capnproto error return values. Rust has it's own way of handling these things so the c++ libmultiprocess code also wouldn't be helpful here.

[ariard]

Thanks for the answers! Yes i'm currently experimenting with capnproto-rust and the other rust wrappers on top of capnproto. I'm able to successfully compile my altnet.capnp by including proxy.capnp and I'll let you know if I meet real hurdles while rewriting the cpp altnet daemons on this branch https://github.com/ariard/bitcoin/tree/2021-07-altnet-lightning in rust ones.

[ryanofsky]

Great! I was looking into possible hurdles calling methods that take proxy ThreadMap or Context arguments like current bitcoin interfaces and example interfaces tend to (like InitInterface.construct method passing ThreadMap, and CalculatorInterface.solveEquation and other methods passing Context). But I think you can ignore these arguments from rust or just not use them in your new interfaces.

But in case you need to use them later:

• The Context.thread member's main purpose is to let remote code execute asynchronously without blocking the capnproto IO thread (eventloop thread). If an IPC method is fast or returns right away without blocking, there's no need for a caller to set this. But if the IPC method is slow or blocking, the IPC caller (client) can first call ThreadMap.makeThread() to create a remote worker thread, and pass the worker thread handle as Context.thread values in subsequent IPC calls so they all execute on the specified worker thread without blocking other IO.

  • Secondarily Context.thread is also used by libmultiprocess to ensure that multiple IPC calls from the same local thread run on the same remote thread. So if one IPC call acquires a lock remotely, and later IPC calls use the lock, and a final call releases the lock, it ensures that all the remote locking, unlocking, and usage happen on the same remote worker thread.

• The Context.remoteThread value is used for more of a corner case. It can be passed so if a local function calls a remote function that needs to call back into a local function within in the same local call stack before the remote call returns, the remote method can pass the Context.remoteThread value which it receives as the Context.thread value it sends when making the callback. This is again useful for locking, so if a local lock is held before making an IPC call which calls back before returning, the callback will be executed in the same thread which already holds the lock.

[ryanofsky]

Another suggestion which might be helpful for working with rust might be to use the https://github.com/bitcoin/bitcoin/pull/19460 branch. This branch adds an -ipcbind option to bitcoin-node so if you run bitcoin-node -ipcbind, on startup it will create a unix socket in <datadir>/sockets/node.sock that rust code can connect directly to. Using the socket, the rust code can call arbitrary Init and get access to other interfaces.

Doing this might be a little easier to start off, because you don't need to change the the c++ code to spawn rust processes.