An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
This is base PR for Fast Drop feature. This is a new implementation of the DROP FEATURE command that requires no waiting time and no history truncation.
The main difficulty when dropping a feature is that after the operation is complete the history of the table still contains traces of the feature. This may cause the following problems:
Reconstructing the state of the latest version may require replaying log records prior to feature removal. Log replay is based on checkpoints, an auxiliary data structure, which is used by clients as a starting point for replaying history. Any actions before the checkpoint do not need to be replayed. However, checkpoints are not permanent and may be deleted any time.
Clients may create checkpoints in historical versions when do not support the required features.
The proposed solution is CheckpointProtectionTableFeature. This is a new writer feature that ensures that the entire history until a certain table version, V, can only be cleaned up in its entirety or not at all. Alternatively, the writer can delete commits and associated checkpoints up to any version (less than V) as long as it validates against all protocols included in the commits/checkpoints planing to remove. We protect against the anomalies above as follows:
All checkpoints before the transition table version are protected. This prevents anomaly (1) by turning checkpoints into reliable barriers that can hide unsupported log records behind them.
Because log cleanup is disabled for the older versions, this also removes the only reason why writers would create new checkpoints, preventing anomaly (2).
This still uses a writer feature, but is a step forward compared to previous solutions because it allows the table to be readable by older clients immediately, instead of after 24 hours. Compatibility with older writers can subsequently be achieved by truncating the history after a 24-hour waiting period.
How was this patch tested?
Added DeltaFastDropFeatureSuite as well as tests in DeltaProtocolTransitionsSuite.
Which Delta project/connector is this regarding?
Description
This is base PR for Fast Drop feature. This is a new implementation of the DROP FEATURE command that requires no waiting time and no history truncation.
The main difficulty when dropping a feature is that after the operation is complete the history of the table still contains traces of the feature. This may cause the following problems:
The proposed solution is
CheckpointProtectionTableFeature. This is a new writer feature that ensures that the entire history until a certain table version, V, can only be cleaned up in its entirety or not at all. Alternatively, the writer can delete commits and associated checkpoints up to any version (less than V) as long as it validates against all protocols included in the commits/checkpoints planing to remove. We protect against the anomalies above as follows:This still uses a writer feature, but is a step forward compared to previous solutions because it allows the table to be readable by older clients immediately, instead of after 24 hours. Compatibility with older writers can subsequently be achieved by truncating the history after a 24-hour waiting period.
How was this patch tested?
Added
DeltaFastDropFeatureSuiteas well as tests inDeltaProtocolTransitionsSuite.Does this PR introduce any user-facing changes?
No.