[Spark] Handle case when Checkpoints.findLastCompleteCheckpoint is passed MAX_VALUE

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

6.98k stars 1.6k forks source link

[Spark] Handle case when Checkpoints.findLastCompleteCheckpoint is passed MAX_VALUE #3105

Closed prakharjain09 closed 1 week ago

prakharjain09 commented 2 weeks ago

Which Delta project/connector is this regarding?

[X] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Fixes an issue where Checkpoints.findLastCompleteCheckpoint goes into an almost infinite loop if it is passed a Checkpoint.MAX_VALUE.

delta-io / delta

[Spark] Handle case when Checkpoints.findLastCompleteCheckpoint is passed MAX_VALUE #3105

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?