Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
2.1k
stars
834
forks
source link
[Bug] PartitionExpire drop partition atomicity when sync to hms #3593
Closed
xuzifu666 closed 1 week ago
Search before asking
Paimon version
master
Compute Engine
spark3.2.0 flink1.17.2
Minimal reproduce step
CREATE TABLE bdsp_test.paimon_mi_24(
user_id
STRING COMMENT '任务用到的库',used_table
STRING COMMENT '任务用到的表',day
STRING COMMENT '按天进行分区' )USING paimon PARTITIONED BY (day);insert into bdsp_test.paimon_mi_24 select 'a', 'a_table', '2024-04-22';
insert into bdsp_test.paimon_mi_24 select 'b', 'a_table', '2024-09-22';
select * from bdsp_test.paimon_mi_24 limit 10;
CALL sys.expire_partitions(table => 'bdsp_test.paimon_mi_24', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd');
If hive version is low which cannot compatible with paimon(such as follow),hms sync would error but paimon drop partition is ok
What doesn't meet your expectations?
Drop partition automicly, If hms failed, paimon also failed.
Anything else?
No response
Are you willing to submit a PR?