apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.1k stars 834 forks source link

[core] Skip pushing down partition filter to file reader #3608

Closed Zouxxyy closed 6 days ago

Zouxxyy commented 1 week ago

Purpose

Skip pushing down partition filter to file reader, for the reasons:

e.g. SELECT * FROM migrate_tbl WHERE pt = 1 always return nothing

Tests

API and Format

Documentation

Zouxxyy commented 6 days ago

Worried about performance regression, maybe we can add a configuration to control this behavior

JingsongLi commented 6 days ago

Worried about performance regression, maybe we can add a configuration to control this behavior

no need, there is no performance regression