apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.43k stars 956 forks source link

[core] Support IN filter pushdown to snapshot/tag/schema system tables #4434

Closed xuzifu666 closed 2 weeks ago

xuzifu666 commented 2 weeks ago

Purpose

Linked issue: close #xxx

  1. Currently IN filter cannot pushdown to snapshot/tag/schema system tables and IN would tramsform to OR(contians all Equal LeafPredicate), if user query with IN filter would cost more unnessary IO in a large number of snapshots/tag/schema;
  2. TagTable predicate specified as LeafPredicate which cannot resolve other kind of Predicate, this pr had improved it.
  3. Add APIs for get snapshots/tags/schemas which get multiple numbers for a list argument.

Tests

API and Format

Documentation