FileReader supports reading data based on specified RowRanges to provide the most fundamental Filter pushdown capability to various upper-level computing engines. This implementation mainly consists of three parts:
Implementing a ChunkBufferedInputStream, which takes ReadRanges as initialization parameters, provides simple IO merging capabilities internally, and shields the upper layer from the discontinuity of IO, ensuring that ColumnReader can read filtered Page data as if reading continuous Pages, achieving the purpose of saving IO.
Implementing the specific data reading based on RowRanges, including interface definitions and the entire data reading pipeline.
Necessary testing and benchmarking.
According to the benchmark results, compared to scanning the entire column chunk, utilizing page pruning yields significant performance improvements. In scenarios with a low number of matched rows, single-column scans exhibit a performance boost of 1 to 30 times, while multi-column scans show an improvement of 10 to 16 times. However, in cases where a larger number of rows are matched, as the number of hit RowRanges increases, the scanning performance gradually deteriorates, potentially even experiencing performance regression. Here are some benchmark test results:
Please kindly request the community's assistance in reviewing and determining whether it can be merged into the community. If needed, I can split the Merge Request into multiple ones. Thank you!
Describe the enhancement requested
FileReader supports reading data based on specified RowRanges to provide the most fundamental Filter pushdown capability to various upper-level computing engines. This implementation mainly consists of three parts:
According to the benchmark results, compared to scanning the entire column chunk, utilizing page pruning yields significant performance improvements. In scenarios with a low number of matched rows, single-column scans exhibit a performance boost of 1 to 30 times, while multi-column scans show an improvement of 10 to 16 times. However, in cases where a larger number of rows are matched, as the number of hit RowRanges increases, the scanning performance gradually deteriorates, potentially even experiencing performance regression. Here are some benchmark test results:
Please kindly request the community's assistance in reviewing and determining whether it can be merged into the community. If needed, I can split the Merge Request into multiple ones. Thank you!
Component(s)
C++, Parquet