apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.1k stars 834 forks source link

[Bug] PartitionExpire drop partition atomicity when sync to hms #3593

Closed xuzifu666 closed 1 week ago

xuzifu666 commented 1 week ago

Search before asking

Paimon version

master

Compute Engine

spark3.2.0 flink1.17.2

Minimal reproduce step

CREATE TABLE bdsp_test.paimon_mi_24( user_id STRING COMMENT '任务用到的库', used_table STRING COMMENT '任务用到的表', day STRING COMMENT '按天进行分区' )USING paimon PARTITIONED BY (day);

insert into bdsp_test.paimon_mi_24 select 'a', 'a_table', '2024-04-22';

insert into bdsp_test.paimon_mi_24 select 'b', 'a_table', '2024-09-22';

select * from bdsp_test.paimon_mi_24 limit 10;

CALL sys.expire_partitions(table => 'bdsp_test.paimon_mi_24', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd');

If hive version is low which cannot compatible with paimon(such as follow),hms sync would error but paimon drop partition is ok 1719229921801.png

What doesn't meet your expectations?

Drop partition automicly, If hms failed, paimon also failed.

Anything else?

No response

Are you willing to submit a PR?