fusio provides Read and Write traits to operate on multiple storage backends (e.g., local disk, Amazon S3) across various asynchronous runtimes—both poll-based (tokio) and completion-based (tokio-uring, monoio)—with:
fusiois now at preview version, please join our community to attend its development and semantics / behaviors discussion.
fusio?In developing Tonbo, we needed a flexible and efficient way to handle file and file system operations across multiple storage backends—such as memory, local disk, and remote object storage. We also required compatibility with various asynchronous runtimes, including both completion-based runtimes and event loops in languages like Python and JavaScript.
fusio addresses these needs by providing:
For more context, please check apache/arrow-rs#6051.
fusio = { version = "*", features = ["tokio"] }fusio supports switching the async runtime at compile time. Middleware libraries can build runtime-agnostic implementations, allowing the top-level application to choose the runtime.
fusio provides two sets of traits:
Read / Write / Seek / Fs are not object-safe.DynRead / DynWrite / DynSeek / DynFs are object-safe.You can freely transmute between them.
fusio has an optional Fs trait (use default-features = false to disable it). It dispatches common file system operations (open, remove, list, etc.) to specific storage backends (local disk, Amazon S3).
fusio has optional Amazon S3 support (enable it with features = ["tokio-http", "aws"]); the behavior of S3 operations and credentials does not depend on tokio.
fusio?Overall, fusio carefully selects a subset of semantics and behaviors from multiple storage backends and async runtimes to ensure native performance in most scenarios. For example, fusio adopts a completion-based API (inspired by monoio) so that file operations on tokio and tokio-uring have the same performance as they would without fusio.
object_storeobject_store is locked to tokio and also depends on bytes. fusio uses IoBuf / IoBufMut to allow &[u8] and Vec<u8> to avoid potential runtime costs. If you do not need to consider other async runtimes, try object_store; as the official implementation, it integrates well with Apache Arrow and Parquet.
opendalfusio does not aim to be a full data access layer like opendal. fusio keeps features lean, and you are able to enable features and their dependencies one by one. The default binary size of fusio is 245KB, which is much smaller than opendal (8.9MB). If you need a full ecosystem of DAL (tracing, cache, etc.), try opendal.
Also, compared with opendal::Operator, fusio exposes core traits and allows them to be implemented in third-party crates.
monoio: all core traits—buffer, read, and write—are highly inspired by it.futures: its design of abstractions and organization of several crates (core, util, etc.) to avoid coupling have influenced fusio's design.opendal: Compile-time poll-based/completion-based runtime switching inspires fusio.object_store: fusio adopts S3 credential and path behaviors from it.