search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.33k stars 684 forks source link

Read Parquet metadata via suffix requests #5979

Open kylebarron opened 1 week ago

kylebarron commented 1 week ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In high-latency environments, like in WebAssembly in users' browsers, minimizing the number of sequential requests can be a significant performance improvement. Now that https://github.com/apache/arrow-rs/pull/5222 has been merged, object_store now has the ability to fetch a suffix byte range of files. It would be great to be able to integrate this with parquet to reduce the number of individual requests required.

Describe the solution you'd like

It seems that parquet::arrow::async_reader::MetadataLoader::load can be refactored to use an initial suffix request instead of needing to know the file size.

Or, perhaps, there should be a new MetadataLoader::load_suffix method, so that implementations can choose to use load if they already know the file size, and use load_suffix if they don't.

Related to this, the ParquetObjectReader::new API requires an ObjectMeta, which requires knowing the file size. It would be great to be able to construct ParquetObjectReader with only the store and the object_store::path::Path. (I'm trying to construct ParquetObjectReader with a fake file length, passing my own ArrowReaderMetadata https://github.com/apache/arrow-rs/pull/5583, but I haven't figured out if that works yet)

Describe alternatives you've considered

Make an extra HEAD request instead of using suffix requests.

Additional context

alamb commented 5 days ago

Or, perhaps, there should be a new MetadataLoader::load_suffix method, so that implementations can choose to use load if they already know the file size, and use load_suffix if they don't.

It seems to me that load_suffix would always be preferred (as it requires less information), except for when the underlying store doesn't support suffix requests.

kylebarron commented 1 day ago

I think the API of AsyncFileReader might have to change? Right now the signature of get_bytes takes a range: core::ops::range::Range<usize>, and presumably that would have to change to something like object_store::GetRange for the concept of a suffix range?

If it does need a breaking API change, does that mean we shouldn't start a PR until 52.2.0 is tagged and released?

alamb commented 1 day ago

If it does need a breaking API change, does that mean we shouldn't start a PR until 52.2.0 is tagged and released?

It is up to you, but I think it would be better to get the PR up and ready (and we can merge it when main opens for 53.0.0 work

alamb commented 1 day ago

Potentially related: https://github.com/apache/arrow-rs/issues/6002