Proposal: (softly) replace Lwt_unix by libuv/uwt

[aantron]

This diagram from the libuv docs sketches the logical structure of libuv. Lwt_unix has basically the same structure.

(source: http://docs.libuv.org/en/v1.x/_images/architecture.png)

However, libuv is maintained by more people, and has a larger user base, so one can expect it to be of higher quality, especially over time. Also, the Lwt Unix binding's Windows side really doesn't get enough attention.

I propose that we eventually somehow "bless" uwt, help maintain it, and/or merge it into Lwt, and start discouraging use of the current Lwt_unix and the rest of the current Unix binding.

uwt already seems to have work-alike modules for most (all?) of the Unix binding, which might make this process easier.

This would, however, introduce a new external dependency.

There are some other possibilities, like mio, but I haven't really looked at them yet. Discussion very welcome.

cc @fdopen

[Drup]

However, libuv is maintained by more people, and has a larger user base, so one can expect it to be of higher quality, especially over time. Also, the Lwt Unix binding's Windows side really doesn't get enough attention.

While that might be true, the same certainly cannot be said about uwt itself.

[c-cube]

This sounds like a lot of work if you don't want to break compat. Lwt_unix maybe hard to maintain (I have looked once or twice upon the internal type definitions for processes, and was surprised by the amount of magic required), but it provides a relativement good and comprehensive API which might not be easy to reproduce. Note I'm mostly interesting in Lwt_process, which provides timeouts and the likes.

Maybe having lwt.uv or lwt.uwt as a new sublibrary would be a proper migration path. Especially if tests that ensure that both lwt.uv and lwt.unix behave the same can be written.

[rgrinberg]

The arguments why a libuv base should be favored are convincing to me but I agree with c-cube that this does not have to mean removing the classical lwt.unix.

I'm also a little concerned about the story of Lwt_engine. Today Lwt_engine seems to be the preferred way of supporting various event loops, but that clearly isn't enough to support something like libuv (at least the way uwt does it). Seems strange to have an API for custom event loops that can't support even Lwt's own needs (libuv support).

[mfp]

On Sun, Mar 26, 2017 at 12:24:09PM -0700, Anton Bachin wrote:

However, libuv is maintained by more people, and has a larger user base, so one can expect it to be of higher quality, especially over time. Also, the Lwt Unix binding's Windows side really doesn't get enough attention.

uwt is to libuv what Lwt_unix is to libev (+ OCaml's Unix on Windows), so the proper comparison is between libev and libuv (and if you wish uwt to Lwt_unix to evaluate the complexity of the corresponding bindings and the likelihood of bugs).

That said yes, libuv is probably already in a better shape nowadays (e.g. IOCP on Windows) and is likely to improve at a much faster rate given the attention it gets thanks to Node.js.

This seems relevant: https://github.com/docker/vpnkit/pull/69

I propose that we eventually somehow "bless" uwt, help maintain it, and/or merge it into Lwt, and start discouraging use of the current Lwt_unix and the rest of the current Unix binding.

uwt already seems to have work-alike modules for most (all?) of the Unix binding, which might make this process easier.

This would, however, introduce a new external dependency.

A merge would be unavoidable I'd say. All current Lwt end-users, except those targeting the browser or node.js (via js-of-ocaml or bucklescript), use Lwt_unix. Keeping uwt in good shape would become the Lwt maintainers' business anyway.

Splitting the Lwt ecosystem in 2/3 wouldn't be a desirable outcome. I believe the only way the move towards libuv could be successful is by pushing functorization (or its moral equivalent with lwt.unix, lwt.uwt ... packages) over Lwt_unix to library developers, and in the process also address https://github.com/ocsigen/lwt/issues/270

(I believe #270 is probably more important than an hypothetical uwt backend itself, for it both allows to use libuv via node.js and would at last bring equal first-class OCaml (atop node.js) support on all platforms, massively improving the OCaml + Win32 experience -- even if on non-windows we'd still want to use the trusty OCaml backend -- cf. the stuff bucklescript is doing on Win32)

This would have to go hand in hand with the development of comprehensive tests for the Lwt_unix interface to both test the new backends and verify compatibility.

So I think some good preliminary steps would be:

• to write such tests (useful whether or not a node.js / libuv backend happens)

• to determine the viability of such "functorization" over lwt.unix (verify whether all the functionality can be mapped as is, or if there are unavoidable differences)

There are some other possibilities, like mio, but I haven't really looked at them yet. Discussion very welcome.

libuv is most likely to stay around (well maintained, and with many eyeballs on it) for a long time, mio is still (too?) young and we don't know yet how successful Rust will be.

-- Mauricio Fernández

[fdopen]

Yes, libuv is basically a c implementation of something very similar to lwt. But there are subtle incompatibilities to Lwt_unix; sometimes because of different aims/design decisions, but more often because they have to wrap iocp under windows, which is not comparable to something like select.

1) select (and epoll/kqueue etc.) inform you, when there was a state change regarding a file descriptor (readable, writable, or just a spurious notice 😛). And then you can do with the file descriptor whatever you want to do. (That's the model abstracted by Lwt_engine).

2) With iocp you ask the operating system to do something and it will inform you, once it has finished the task.

The first model is far more friendly to language likes OCaml. You can wait until something changed with the states of your file descriptors and then you can do most of the work inside Ocaml. The second model forces you to move your data (buffers,etc.) outside the stack and OCaml heap. As long as the operation is not finished, you should obviously not move your buffers, etc. around.

A further consequence of this is that under uwt most threads are not cancelable. With select you can just ignore any further state changes, whenever Lwt.cancel was called. With iocp the operating system will perform the requested operations in parallel. When you decide to cancel it, it might already be partially or fully completed. Therefore something like Lwt.cancel is largely missing under libuv. Cancellation only works for tasks that are queued for the threadpool and are not yet running.

And because libuv tries to abstract away such different mechanisms, things are often more obscure. Something missing under lwt.unix? It's usually easy to add. It might not work under windows or only under linux, but that often doesn't matter. Trying to add something under libuv might not be that easy. You have to reverse the abstraction layer in mind, then rely on things that are not covered by the official API and then pray that upstream libuv won't introduce a breaking change with the next release. But of course it depends on what you have in mind. Most things are probably easy to cover with either http://docs.libuv.org/en/v1.x/poll.html or http://docs.libuv.org/en/v1.x/threadpool.html

And yes, higher level modules like Lwt_io, Lwt_process,etc. usually have their counterparts inside uwt - with nearly identical names and interface. And what's done in system threads, is also available under uwt with a nearly identical interface.

[bikallem]

There's also this https://github.com/piscisaureus/wepoll It seems to advertise itself as a drop in replacement for Linux epoll in windows. This could maybe provide a better migration path/alternative to more performance windows support.

[rgrinberg]

@bikallem libuv solves many more problems than having a cross platform epoll. For example it has an asynchronous, cross platform DNS implementation.

One pre-req for using Uwt in Lwt should be porting Uwt to dune btw. Another point to consider is the huge amount of C code that's currently in Uwt. I have quite a lot of faith in the competence of the author of Uwt, but we should really look into minimizing the amount of C code through things like Ctypes and automatic binding generation.

[aantron]

Should be more feasible now with https://github.com/aantron/luv.