[Standalone] Safer checkpointing and better logging

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.62k stars 1.71k forks source link

[Standalone] Safer checkpointing and better logging #3646

Closed scottsand-db closed 2 months ago

scottsand-db commented 2 months ago

Which Delta project/connector is this regarding?

[ ] Spark
[X] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

This PR

adds better logging throughout Delta Standalone (improves debuggability)
fixes an issue in the Delta Standalone Checkpoint code path where we would incorrectly close an incomplete checkpoint upon error

How was this patch tested?

Existing UTs + tested and verified locally

delta-io / delta

[Standalone] Safer checkpointing and better logging #3646

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?