Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
2.1k
stars
834
forks
source link
[core] PartitionExpire drop partitions consisitent with hms drop #3594
Closed
xuzifu666 closed 1 week ago
Purpose
PartitionExpire drop partition atomicity when sync to hms. CREATE TABLE bdsp_test.paimon_mi_24( user_id STRING COMMENT '任务用到的库', used_table STRING COMMENT '任务用到的表', day STRING COMMENT '按天进行分区' )USING paimon PARTITIONED BY (day);
insert into bdsp_test.paimon_mi_24 select 'a', 'a_table', '2024-04-22';
insert into bdsp_test.paimon_mi_24 select 'b', 'a_table', '2024-09-22';
select * from bdsp_test.paimon_mi_24 limit 10;
CALL sys.expire_partitions(table => 'bdsp_test.paimon_mi_24', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd');
If hive version is low which cannot compatible with paimon(such as follow),hms sync would error but paimon drop partition is ok
Linked issue: https://github.com/apache/paimon/issues/3593
Tests
API and Format
Documentation