[flink] PartitionMarkDone enables the use of various partition trigge…

[Aitozi]

…r strategies.

Purpose

Linked issue: close #xxx

In our company, we have encountered an issue with the HMS partition statistic being incorrect. This is because during the writing process, we only update the metastore partition when it is first written to. Therefore, we would like to implement the PartitionMarkDone strategy to update the statistics in HMS after a short idle period for each partition. We need a separate configuration for PartitionMarkDone due to differing requirements:

• We do not want to rely on extracting accurate timestamps from partition names.
• We aim to store each partition's last access time in state so that it can be reported back to HMS. To achieve this, we plan on extending PartitionMarkDone functionality to support different trigger strategies for partitions. Following this extension, we will be able to introduce custom triggers such as PartitionHmsReporterTrigger.

Tests

API and Format

Documentation

[Aitozi]

WDYT? @JingsongLi

[JingsongLi]

Hi @Aitozi , can you explain this pr from API level? Or you are modifying current behavior?

[Aitozi]

@JingsongLi The behavior is not change. In this PR, I extract the PartitionTrigger interface, The default is PartitionMarkDoneTrigger.

In StoreCommitter, it acts the PartitionCollector, it can trigger based on different config such as partition-mark-done or hms-report