search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[flink] Introduce partition mark done with batch #3507

Closed wwj6591812 closed 3 months ago

wwj6591812 commented 3 months ago

Purpose

Base on https://github.com/apache/paimon/pull/3317

This PR is intended to support a mechanism for marking a partition as finished and subsequently pulling up offline scheduling jobs with flink batch job.

Tests

API and Format

Documentation

JingsongLi commented 3 months ago

We can unify batch and streaming to one trigger.

wwj6591812 commented 3 months ago

yes, updated，pls cc, thx