search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.52k stars 749 forks source link

Improve page index metadata loading in `SerializedFileReader::new_with_options` #6491

Open alamb opened 2 days ago

alamb commented 2 days ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. As @etseidl pointed out in https://github.com/apache/arrow-rs/pull/6466/files#r1778966728

This should be a bit more efficient since read_page_indexes will fetch the necessary bytes from the file in a single read, rather than 2 reads per row group.

We can use the new ParquetMetaDataLoader API to read the page indexes in more efficiently (fewer IOs for example)

However, when I tried to implement it, we caught what appears to be a subtle bug -- specifically that the predicates would have been ignored: https://github.com/apache/arrow-rs/pull/6466/files#r1783526090 -- no tests failed.

Describe the solution you'd like

I would like to:

  1. Reduce the IO's needed to read page indexes in SerializedFileReader::new_with_options, and clean up the code to use the new ParquetMetaDataReader
  2. Add test coverage for reader predicates and page index

Describe alternatives you've considered leave as is

Additional context

etseidl commented 1 day ago

@alamb are you working on this or would you like me to take it?

alamb commented 18 hours ago

@alamb are you working on this or would you like me to take it?

I am not planning to work on this at this time. If you had time to look at it that would be great.

THank you 🙏

etseidl commented 14 hours ago

take