Calling Lwt Ocaml code from Rust

[emchristiansen]

If possible, please support calling Lwt Ocaml code from Rust. If there's already a good workflow for this, please describe it in the book. If this isn't something you plan to support, please note that in the book.

The best I can do right now is to run every Lwt function I wrap through Lwt_main.run, but I suspect there's substantial overhead when doing this for each and every function.

Thanks!

[zshipko]

Thanks for the suggestion! Suppprting Lwt is a great idea, in the past I have used Lwt_main.run since performance wasn’t the main concern, but it would be interesting to see how to improve on that a bit!

Instead of adding Lwt support directly to ocaml-rs I could see it existing as a separate Rust crate.

[tizoc]

When I investigated this some time ago (didn't end implementing anything in the end because I switched to working on other stuff), what I found was that probably the best way was to expose Lwt's "job" API to Rust, probably with some helper macros.

Here is an example in C that wraps getcwd in a way that makes it asynchronous and friendly to Lwt:

https://github.com/ocsigen/lwt/blob/master/src/unix/unix_c/unix_getcwd_job.c

[Lupus]

PyO3 project (integration between Rust and Python) has support for binding async Rust code to Python and vice versa [1]. Lwt's "job" api seems quite heavy-weight, it will result in the thread pool waiting for C function to complete. Would be cool to have some lightweight mapping of Rust async functions to Lwt promises and vice versa like PyO3 is doing.

Rust ecosystem is a very promising source of batteries for OCaml. With Multicore OCaml we're even closer to Rust.

@raphael-proust maybe you have some ideas how Lwt can expose stable API for managing of promises from some C code? Maybe some parts of "job" API can be exposed directly?

cc @talex5 as integration with Rust might be even more interesting for Eio.

[1] https://pyo3.rs/v0.17.3/ecosystem/async-await

[emchristiansen]

It looks like there was some interest in pyml in doing something similar, though I don't think they found a solution.

[raphael-proust]

I don't know how the rust-ocaml bindings work so I might not be able to offer much help.

The main draw back of calling Lwt_main.run around each OCaml+Lwt function call is that you lose the ability to do two such calls concurrently. But I don't think there would be a massive overhead in terms of additional cost: it's more of a missed opportunity of concurrency.

---

Here's something to try out. It might not work, but if it doesn't then it might inform us how to make progress on this issue:

From rust, call OCaml. Specifically call an OCaml function which calls Lwt_main.run and from inside Lwt_main.run calls back (rather than returns) into rust.

Then from rust you can call Lwt functions which should (:crossed_fingers:) be running "inside the scheduler".

Those calls to Lwt functions would return promises which are just normal OCaml values which can be manipulated normally. One thing that can be done is attach callbacks to those promises. One thing those callbacks could do is call into rust (as a way to notify completion of some Lwt task).

---

I can assist into this investigation but I cannot lead it: I don't know enough about the cross-language calls. Don't hesitate to ping me if you have questions and such. Also don't hesitate to ping me if you want to set up a collaborative hacking session.

[zshipko]

Thank you for this insight - that experiment sounds like a great idea! I will get that set up and give an update here

[emchristiansen]

@zshipko Any luck?

[zshipko]

This is as far as I got: https://gist.github.com/zshipko/b5e91613621b9b71e2f70dc2ca1552d3

This uses Lwt.poll inside Rust's Future::poll to resolve the promise. I'm still not sure how this would be used but feel free to take the code if you want to do continue research on this topic - I probably won't have time to work on this for a while.

[emchristiansen]

Zach, I think your sketch was almost there. Here's what I came up with.

I think the biggest remaining issues are:

1. Busy polling on the Rust side, because I don't know how to thread through the "waker" signal from OCaml.
2. There's no type-level distinction between Lwt and non-Lwt types in Rust (see the TODO).

[Lupus]

Wow, looks nice! Maybe some eventfd could help pass the signal from OCaml to Rust event loop? 🤔

[emchristiansen]

@Lupus Do you have a particular design in mind?

[Lupus]

When you need to pass Lwt.t to Rust, you instead pass some wrapper object, that includes eventfd, and a value for promise resolution/rejection, encoded as Rust's result for example. Probably you'll also need a flag indicating if some value for this promise is still being awaited. Then you add a hooks on OCaml side to resolution/rejection of the promise and send event via eventfd to Rust side. Rust side gets the wrapper object, checks the flag, if it needs to wait - adds eventfd handle to whatever event loop is being used. Once event arrives, recheck the flag and if everything is ready - fetch the resolution/rejection value from the wrapper.

Probably this wrapping and notifying can be implemented in Rust and exported to OCaml as some API, and ocaml-rs can come with some macros to wrap promises being passed to Rust side from OCaml.

The reverse direction could probably work with Lwt notification API:

Thread-safe notifications

If you want to notify the main thread from another thread, you can use the Lwt thread safe notification system. First you need to create a notification identifier (which is just an integer) from the OCaml side using the Lwt_unix.make_notification function, then you can send it from either the OCaml code with Lwt_unix.send_notification function, or from the C code using the function lwt_unix_send_notification (defined in lwtunix.h).