[Feature] Support hidden partition like Iceberg

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.28k stars 912 forks source link

[Feature] Support hidden partition like Iceberg #3370

Open eric666666 opened 3 months ago

eric666666 commented 3 months ago

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation

Iceberg's hidden partition is easy to use which prevents user mistakes that cause silently incorrect results or extremely slow queries. And partition key can be transformed from existed column.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

[X] I'm willing to submit a PR!

JingsongLi commented 3 months ago

This Iceberg mechanism can affect a lot of code at a very high cost.