Switch to kqueue on Darwin for asynchronous I/O

penberg / limbo

Limbo is a work-in-progress, in-process OLTP database management system, compatible with SQLite.

MIT License

602 stars 38 forks source link

Switch to kqueue on Darwin for asynchronous I/O #25

Closed penberg closed 2 days ago

penberg commented 5 months ago

Alternative to https://github.com/penberg/limbo/issues/22

aneessh18 commented 2 months ago

Hey @penberg, I've just read the paper on the architecture of limbo and I want to contribute to it. Do you mind expanding on the goal of this issue?

penberg commented 2 months ago

Hey @aneessh18! The idea is to hack core/io/darwin.rs to use something like https://crates.io/crates/kqueue to submit I/O asynchronously instead of using the synchronous File::read_exact() API.

aneessh18 commented 2 months ago

@penberg So I've looked into the APIs offered by the kqueue lib and it seems like a library to monitor the events pertaining to a file. I wasn't able to find how to leverage kqueue to read a file asynchronously. But I was able to find enough documentation on how to read a file async using tokio. Is that appropriate ? ref: https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/FSEvents_ProgGuide/KernelQueues/KernelQueues.html

penberg commented 2 weeks ago

@aneessh18 What you want to do is:

(1) Open file in non-blocking mode (for example with rustix)
(2) Read in DarwinFile::pread(). If the read succeeds, complete the completion and bail out; otherwise go to (3).
(3) Add a watch to kqueue when the file descriptor becomes readable.
(4) In DarwinIO::run_once(), poll for events, when file descriptor is readable, read and complete completion.

It's worth considering using https://crates.io/crates/polling crate because that gives #41 for free.

penberg commented 2 days ago

Fixed by https://github.com/penberg/limbo/pull/133