[Spark] Throw exception when additional listing also doesn't reconcile the listing gap in testing

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.22k stars 1.62k forks source link

[Spark] Throw exception when additional listing also doesn't reconcile the listing gap in testing #3244

Open sumeet-db opened 2 weeks ago

sumeet-db commented 2 weeks ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

We already log the failure scenario when additional file-system listing also can't reconcile the gap between concurrent file-system listing and commit-owner calls.

With this PR, we will throw an exception if the above condition is triggered while testing.

How was this patch tested?

UTs

delta-io / delta

[Spark] Throw exception when additional listing also doesn't reconcile the listing gap in testing #3244

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?