search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[core] Fix the expired partition sync to metastore #3534

Closed gsralex closed 3 months ago

gsralex commented 3 months ago

Purpose

Linked issue: close When metastore.partitioned-table is enabled, the partition information in hms will not be deleted after the expired partition is automatically deleted.

Tests

API and Format

Documentation

leaves12138 commented 3 months ago

Thanks for @gsralex, can you run mvn spotless:apply to fix the build issue after adding some tests for expired partition function you have written?

gsralex commented 3 months ago

Thanks for @gsralex, can you run mvn spotless:apply to fix the build issue after adding some tests for expired partition function you have written?

OK, I'm very willing.

leaves12138 commented 3 months ago

Can you write a test for this? You can refer to HiveCatalogITCaseBase.testFlinkWriteAndHiveReadToCompare. Use flink tEnv to write data and expire partition, use hiveShell to check if the partition is dropped actually in hive metastore.