Open clbarnes opened 7 months ago
Async traits are stable, and object-store has a release candidate with suffix requests, so this can get going.
Some design decisions tie in to the async strategy.
object-store asynchronously provides the whole block of bytes rather than a reader. IRL anything providing a Read/ Write interface will probably be doing so over an opaque buffer anyway, and chunks should be small compared to RAM so I don't think this is a problem. The difference is how that buffer is allocated, I suppose.
Async functions should not block, i.e. all CPU-heavy functions (e.g. compression) should be pushed down into async functions. But at the end of the day, these are probably just going to call spawn_blocking
, which has its own overhead, especially if we're doing lots of small reads, as we expect to be. We like the async nature of the original fetch, because there's waiting involved, but don't get a lot from async read/write through the codec chain. It might be better, therefore, to have CPU-heavy blocking code inside the (async) codec chain, with the expectation that the caller tucks each chunk's codec chain into a spawn_blocking
to parallelise it at the top level (fetching multiple chunks to make a multi-chunk read).
@JackKelly has probably thought about concurrency models as they apply to IO more than I have - any thoughts? The options are
So we'll keep the current traits (with some signature changes) and just impl<S: ObjectStore> (Readable/Listable/Writeable)Store for S
.
Hey Chris! This all sounds very interesting.
FWIW, I've been slowly hacking away on light-speed-io. I'm currently focused on using io_uring
to fetch large numbers of chunks from local disks. light-speed-io
isn't a Zarr implementation. Instead the aim is to provide an easy-to-use API for fetching large numbers of chunks of files concurrently, using different optimisations for different storage systems. I'm toying with the idea of having light-speed-io
take a user-supplied function, so light-speed-io
can also orchestrate (and parallelise) the processing of each chunk. Or maybe light-speed-io
will just expose an iterator of loaded chunks, and it'll be entirely the user's responsibility to
On the topic of async: I was originally thinking of using async. But I've shifted towards using rayon
to process chunks in parallel. And all the async IO will be handled by the operating system (using io_uring
). Here's some very hacky proof-of-concept rust code using io_uring
and Rayon.
One day, I'd hope that light-speed-io
could be the IO backend for zarr3-rs. But that day isn't today! It'll be months before light-speed-io
does anything very useful, and even longer before light-speed-io
has mature support for cloud storage, so I wouldn't want to block your work!
https://crates.io/crates/object_store
Pros
Cons
We use traits a lot, and async methods on traits aren't stable yet, but may be as of 2023-12-28get
andget_range{s}
use an async stream wrapping aBytes
respectively; both would need wrapping overdoesn't allow suffix ranges yet https://github.com/apache/arrow-rs/issues/4611With any move to async, BB codecs in particular would need major rewrites, and make use of https://crates.io/crates/async-compression for compression.