apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[core] Add an expiration strategy that comparing with partition updat… #3664

Closed LinMingQiang closed 2 months ago

LinMingQiang commented 3 months ago

…e time.

Purpose

Add an expiration strategy that comparing with partition update time.

Using partition.expiration-strategy option to specifies the expiration strategy for partition expiration.

Possible values:

VALUES-TIME (default): A partition expiration policy that compares the time extracted from the partition value with the current time.

UPDATE-TIME : A partition expiration policy that compares the last update time of the partition with the current time.

What is the value of this feature :

Linked issue: close #xxx

Tests

API and Format

Documentation

JingsongLi commented 3 months ago

Some discussion: If the partition conversion time fails, I feel like we can throw an exception and remind user to configure the formatter or use update-time mode.

LinMingQiang commented 3 months ago

Changing the expiration strategy for a table with data is risky, may be we could provide a procedure to expire the specified wrong partition.