The port to Eio of libirmin inherited a defect from the previous integration with the Lwt scheduler: Every calls from C into OCaml land would start a fresh scheduler, run the Irmin function, stop the scheduler and return the result to the C world. Not only is it inefficient to start/stop the scheduler so often, stopping Eio will also close any opened database files thanks to its strict resource management... which was not an issue before as Lwt allowed us to leak the open files across calls!
To address this, @talex5 recommended that we spawn a new domain to run the Eio scheduler on, with the C thread submitting request to it and blocking until the response comes back... but since libirmin doesn't expose any "async" operations to C, we would always have a single domain active at a time (either the C thread or the Eio one).
So it was slightly easier and more fun to flex some effect handlers to achieve the same outcome without a second domain:
We start the Eio scheduler once on the first C call to an Irmin function, but we don't let Eio stop as we suspend it by performing an effect and save its continuation for later
The next C call to Irmin will resume the suspended Eio scheduler, just long enough to run the Irmin function in its scope, then suspend the scheduler again, etc
cc @kayceesrk "Eio dawg, I heard you like schedulers"
kayceesrk
commented
3 months ago
(builds on top of the other Eio PRs, only the last commit https://github.com/mirage/irmin/commit/121309c2a25ab10d9de30acba63d5a16823b5268 is new)
The port to Eio of
libirmininherited a defect from the previous integration with the Lwt scheduler: Every calls from C into OCaml land would start a fresh scheduler, run the Irmin function, stop the scheduler and return the result to the C world. Not only is it inefficient to start/stop the scheduler so often, stopping Eio will also close any opened database files thanks to its strict resource management... which was not an issue before as Lwt allowed us to leak the open files across calls!To address this, @talex5 recommended that we spawn a new domain to run the Eio scheduler on, with the C thread submitting request to it and blocking until the response comes back... but since
libirmindoesn't expose any "async" operations to C, we would always have a single domain active at a time (either the C thread or the Eio one). So it was slightly easier and more fun to flex some effect handlers to achieve the same outcome without a second domain:cc @kayceesrk "Eio dawg, I heard you like schedulers"