apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.1k stars 834 forks source link

[core] PartitionExpire drop partitions consisitent with hms drop #3594

Closed xuzifu666 closed 1 week ago

xuzifu666 commented 1 week ago

Purpose

PartitionExpire drop partition atomicity when sync to hms. CREATE TABLE bdsp_test.paimon_mi_24( user_id STRING COMMENT '任务用到的库', used_table STRING COMMENT '任务用到的表', day STRING COMMENT '按天进行分区' )USING paimon PARTITIONED BY (day);

insert into bdsp_test.paimon_mi_24 select 'a', 'a_table', '2024-04-22';

insert into bdsp_test.paimon_mi_24 select 'b', 'a_table', '2024-09-22';

select * from bdsp_test.paimon_mi_24 limit 10;

CALL sys.expire_partitions(table => 'bdsp_test.paimon_mi_24', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd');

If hive version is low which cannot compatible with paimon(such as follow),hms sync would error but paimon drop partition is ok

1719229921801.png

Linked issue: https://github.com/apache/paimon/issues/3593

Tests

API and Format

Documentation