[common][core][format] Support using file index result for RowGroup f… - Githubissues

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.43k stars 956 forks source link

[common][core][format] Support using file index result for RowGroup f… #4470

Closed hang8929201 closed 2 weeks ago

hang8929201 commented 2 weeks ago

Purpose

Use file index result(bitmap) for more fine-grained IO clipping in the reader.

Tests

This is a query in our real business scenario.

no index:
- index for parquet block filtering:
- index for orc rowgroup filtering:

API and Format

Documentation