[flink] Optimize watermark check in SnapshotManager which can earlier quit reduce FileIO with Filesystem - Githubissues

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.1k stars 834 forks source link

[flink] Optimize watermark check in SnapshotManager which can earlier quit reduce FileIO with Filesystem #3635

Closed xuzifu666 closed 3 days ago

xuzifu666 commented 3 days ago

Purpose

Optimize watermark check in SnapshotManager which can earlier quit reduce FileIO with Filesystem: we can find many watermark in snapshot could be Long.MIN firstly which should not be compute.

Linked issue: close #xxx

Tests

API and Format

Documentation