search

apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
474 stars 97 forks source link

feat: Extract FileRead and FileWrite trait #364

Closed Xuanwo closed 1 month ago

Xuanwo commented 1 month ago

As part of https://github.com/apache/iceberg-rust/issues/356, this PR extracts the FileRead and FileWrite traits.

With this PR, we no longer depend on futures::AsyncRead, futures::AsyncWrite, tokio::io::AsyncRead, or tokio::io::AsyncWrite. Instead, we've introduced our own IO trait as a bridge to implement upstream traits directly, reducing overhead. For instance, by implementing parquet's AsyncFileRead directly, we eliminate the need to copy bytes just to satisfy futures::AsyncRead.

Additionally, the new design will enable us to optimize further in ways that align more closely with our specific business logic.

Bonus: Our new IO traits are both completion-based I/O and vectored I/O ready. We can implement these optimizations once we finalize the API we want to use.

Xuanwo commented 1 month ago

cc @Fokko @liurenjie1024 @ZENOTME for reviews.