[Spark] Conditionally check for presence of delta file at checkpoint version

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.17k stars 1.62k forks source link

[Spark] Conditionally check for presence of delta file at checkpoint version #3080

Open sumeet-db opened 1 month ago

sumeet-db commented 1 month ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

If validateLogSegmentWithoutCompactedDeltas is set, check that we have a delta file at the checkpoint version if there are no newer versions.

How was this patch tested?

UTs

delta-io / delta

[Spark] Conditionally check for presence of delta file at checkpoint version #3080

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?