An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://github.com/delta-io/delta/pull/3799 added the capability to write a Checksum file after every commit. However, writing a checksum currently requires a full state reconstruction --- which is expensive. This PR adds the capability to compute most of the fields incrementally (apply the current delta on top of the last checksum to get the checksum of the current version). This works as long as the the actual operation performed matches exactly with the specified operation type in the commit. Note that this feature is gated behind a flag that is true by default.
Which Delta project/connector is this regarding?
Description
https://github.com/delta-io/delta/pull/3799 added the capability to write a Checksum file after every commit. However, writing a checksum currently requires a full state reconstruction --- which is expensive. This PR adds the capability to compute most of the fields incrementally (apply the current delta on top of the last checksum to get the checksum of the current version). This works as long as the the actual operation performed matches exactly with the specified operation type in the commit. Note that this feature is gated behind a flag that is
trueby default.How was this patch tested?
Added tests in ChecksumSuite.
Does this PR introduce any user-facing changes?
No