Windows support

[talex5]

Eio needs to support Windows. It was somewhat working with the libuv backend, but the other platforms have stopped using that now (see #434).

• [x] We need OCaml 5.1 for https://github.com/ocaml/ocaml/pull/11846 (OCaml 5.0 segfaults on Windows)

The current plan is:

• [x] Create an eio_windows backend based on eio_posix (#497).
• [x] Get eio_main working with Windows. Get the CI to run one of the examples.
• [ ] Set up CI to run the main set of tests on Windows.
• [x] Add initial FS support (#509).
• [ ] Finish FS support (see https://github.com/ocaml-multicore/eio/pull/509#issuecomment-1573430005).
• [ ] Need to decide about absolute paths (/C:/foo/bar?) and converting from native paths.
• [ ] Go through the backend and implement all the TODOs.
• [ ] Improve timer resolution (currently 1 second).

Later, we could convert the backend to use IOCP, but that's not ready yet.

[nojb]

Hiya, just FYI I am beginning to try this on Windows. I managed to build the library with the mingw64 compiler and the libuv backend (not completely trivial due to the many dependencies :)). A trivial test program

let () =
  Eio_luv.run @@ fun env ->
  let clock = Eio.Stdenv.clock env in
  Eio.traceln "The time is now %f" (Eio.Time.now clock)

fails with

Fatal error: exception Invalid_argument("Sys.signal: unavailable signal")

Fix is probably easy (just ignore signals?) but anyway, I don't think you should run to fix this one issue; I am planning to experiment a bit more and will report back (adding to this issue).

[talex5]

Ah, great! Yes, it probably just needs this disabled:

https://github.com/ocaml-multicore/eio/blob/8beda18c9950aa8f0230b9f699d47b8d7bfe214a/lib_eio_luv/eio_luv.ml#L28

(OCAMLRUNPARAM=b should give the location if not)

[nojb]

BTW what do you use for testing? The files in test/?

[talex5]

Yes. dune runtest should run all the tests.

make bench runs the benchmarks.

[nojb]

Yes. dune runtest should run all the tests.

Unfortunately they rely on mdx and its status on Windows is not very clear (see realworldocaml/mdx#295). I will try to run them manually.

[talex5]

Ah, is this because of different line-endings? It should be possible to configure Git to check the files out with Unix line endings, for testing. It looks like we're not using any shell blocks, so that should be OK.

[polytypic]

I'm looking into getting Eio working on Windows. I was just able to reproduce the signal issue:

C:\...\eio>make bench
dune exec -- ./bench/bench_buf_read.exe
+Read 100000000 bytes in 0.875s
dune exec -- ./bench/bench_mutex.exe
Fatal error: exception Invalid_argument("Sys.signal: unavailable signal")
make: *** [Makefile:8: bench] Error 2

[polytypic]

Some progress on Windows: After removing the set_signal call from eio_luv.ml, the benchmarks started to fail with a segfault. The cause of the segfault turned out to be a compiler bug, see PR. I have verified that with the PR the segfaults from benchmarks are gone.

I plan to continue with trying to get mdx working on Windows for running tests.

There also appears to be at least one Obj.magic call in eio_luv.ml that will not work on Windows.

[talex5]

That's excellent progress!

BTW, there are now some examples you can run that will test libuv and don't need MDX (though getting MDX working so we can run the tests is still important, of course):

$ dune exec -- ./examples/hello/main.exe
Hello, world!

$ dune exec -- ./examples/net/main.exe
+client: Connecting to server at tcp:127.0.0.1:8080...
+server: Accepted connection from tcp:127.0.0.1:38524
+server: Received: "Hello from client"
+client: Got reply "OK"

[patricoferris]

I was wondering if it might be useful to have a more substantial set of tests that don't rely on MDX? There are two (related) uses cases for them in my mind:

• Adding a new backend e.g. GCD. Tests are really useful to help get the implementation right, but MDX doesn't offer a particularly friendly interface for running a single test or debugging when I inevitably get the implementation wrong.
• The Javascript backend (PR coming soon...) won't work with MDX and it would be nice to have a suite of tests we can run (Alcotest for example supports running in Node and can support the browser too).

If there's a desire for this I can open a separate issue, just saw this comment so replied here.

[talex5]

Yes, it would be nice if MDX made debugging easier! It does have --section, which might help. But dune runtest doesn't pass arguments though (it doesn't for other test frameworks either). Also, I usually put some helpers at the start, so it would need to know to run that bit in all cases.

As a work-around, I sometimes indent the bit of the file I don't want to test with 4 spaces, so MDX ignores it. Or use dune runtest &| head -$LINES to see the first error instead of the last (note: this is fish syntax).

I guess it would be fairly easy to get MDX to generate stand-alone expect tests from the .md files. ocaml-mdx pp file.md already generates OCaml code, but it doesn't include the expected output.

[polytypic]

Just noting some progress here.

First of all a PR was reverted in mdx that broke it on Windows.

I've also made some progress towards making it possible to use mdx tests with the luv backend on Windows. The problem is that, on Windows, DLLs need to be created with flexlink in order to be able to load them in the toplevel. More work is still needed to prepare a proper PR to luv to build the vendored library properly.

[avsm]

It increasingly looks like we'll switch away from luv (on all platforms: #434), and support Windows directly using iocp. Is this a good time to modify this issue to reflect that, and to get a status update on where we stand with iocp support?

The first step is to release direct bindings to iocp, the latest branch of which is with @djs55: https://github.com/djs55/ocaml-iocp/commits/bench and are pretty stable.

[talex5]

Is this a good time to modify this issue to reflect that, and to get a status update on where we stand with iocp support?

Good idea. I've updated the summary. @polytypic has been working on iocp recently too.

[avsm]

@polytypic, glad to hear you've been hacking on iocp too. It would be useful to compare notes with what your trees and modifications look like with the one I posted above.

[samwgoldman]

In #434 @avsm asked:

Just to help scope out your use of Windows here, are you primarily interested in a CLI to generate a flow.exe, or other Win32 API integration points?

We distribute a Flow binary for Flow and support building for Windows from source. I'm interested in a reliable Windows concurrency library to replace our current usage of Lwt.

Currently, we use Lwt with a select-based backend, not luv, but a IOCP-based implementation which is well-tested would be fantastic. We'd happily switch to Eio for it.

[avsm]

Thanks @samwgoldman, that's in scope for what we're doing with Eio for the first version. I was a little worried there would be a need for Win32 GUI libraries and such, but a CLI tool should be entirely possible to replace Lwt->Eio with (and potentially even speed things up).

[avsm]

An update: I'm continuing to clean up the IOCP bindings in https://github.com/avsm/ocaml-iocp/tree/main/src (now with working CI!). There were two approaches to handling buffers in the branch I took up work on, and there still remains some integration that needs to happen there before it can be integrated with eio (specifically, deciding between using uring's Heap vs the static array of Overlapping buffers).

I'm travelling till the 10th April, so won't make any progress until after then. I estimate it's just a day or two's work to get the bindings into suitable shape to have a Eio_windows backend (for simple filesystem/networking). If anyone wants to take over in the next few weeks, feel free; I'll catch up when back!

[avsm]

Incidentally, I don't have any views on "Make sure the cross-platform API is OK for windows", so #124 would be an excellent topic for anyone interested in advancing Windows to give input/PRs on, @samwgoldman (and others).

[patricoferris]

I might take a look at bringing my old eio_iocp backend up to speed with the latest improvements and fixes that the IOCP bindings have undergone (+ with some more experience I've had with Eio backends), just wondering if others are doing the same to avoid stepping on any toes ? If there is a dev branch somewhere also happy to take it for a spin and help with reviewing :))

[djs55]

I’ve not had the time to look at eio so far but I’m keen to try anything you come up with!

[avsm]

I can also test an eio_iocp backend out pretty quickly -- I have a Windows/OCaml dev environment available again.

[talex5]

At the moment, the eio_windows code is just failwith "TODO: Windows support.", and I'm fairly happy to merge anything that improves on that! For example, if you have an equivalent to lib_eio_linux/sched.ml we could get that merged now, even if the rest of the code is missing, and let other people contribute the remaining bits. I know @polytypic is planning to work on this too, but I hear you're already coordinating with him to avoid duplicated work.

[polytypic]

~I just got this PR to build and test on my Windows machine. I'll try to take a moment to read through the changes today.~

Sorry, wrong discussion thread. 😅

[patricoferris]

There are some TODOs from the ecosystem point of view:

• [ ] Upstream Windows compilers to opam-repository from https://github.com/dra27/opam-repository/tree/windows-5.0
• [x] Fix the ocplib-endian problem, in fact crowbar doesn't actually need it so that route might be easier and removes an extra dependency which is always nice https://github.com/stedolan/crowbar/pull/67
• [x] Add Cstruct-based Unix file operations which everyone can benefit from https://github.com/mirage/ocaml-cstruct/pull/302

[polytypic]

Noting here that the "symlinks" test was failing on my Windows machine although I could create symlinks from the command line. Looking at the symlink creation code in the ocaml/otherlibs, I noticed that it uses SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE if the registry key SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock has the DWORD value AllowDevelopmentWithoutDevLicense set to 1. After creating that value in the registry the symlinks test passed.

[Chimrod]

Hello, how should I report missing function in Eio.Path?

I have an error when using eio in windows (preadv is not supported on windows yet) and I would like to track or report this issue. Should I open another one issue or do you want to keep all thoses reports inside this thread?

[talex5]

how should I report missing function

I don't think they particularly need reporting - it's easy to find the unimplemented stubs in eio_windows_stubs.c (though some // TODO comments could be added). Mostly, we just need some Windows dev to implement them!