search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.1k stars 834 forks source link

[Feature] Optimize watermark check in SnapshotManager which can reduce FileIO with Filesystem #3637

Closed xuzifu666 closed 3 days ago

xuzifu666 commented 3 days ago

Search before asking

Motivation

Reduce File IO with Filesystem when all watermark in snapshots.

1719634609775.png

Solution

early quit when all watermark is Long.MIN

Anything else?

No response

Are you willing to submit a PR?