search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Spark] Pass table identifier from DeltaLog constructor to commit coordinator #3718

Closed ctringdb closed 1 month ago

ctringdb commented 1 month ago

Which Delta project/connector is this regarding?

Description

Propagate the table identifier through the following code paths. This parameter is set to None at the moment. The call sites will be fixed in subsequent PRs.

DeltaLog constructor
    ⇒ getSnapshotAtInit
        ⇒ createLogSegment (protected)
            ⇒ listDeltaCompactedDeltaAndCheckpointFiles
                ⇒ listDeltaCompactedDeltaCheckpointFilesAndLatestChecksumFile
                    ⇒ ...complete path to the commit coordinator...
            ⇒ getLogSegmentForVersion
                ⇒ getLogSegmentWithMaxExclusiveCheckpointVersion
                    ⇒ listDeltaCompactedDeltaAndCheckpointFiles
                        ⇒ [see above]
        ⇒ getUpdatedSnapshot
            ⇒ createLogSegment (private)
                ⇒ createLogSegment (protected)
                    ⇒ [see above]

How was this patch tested?

Unit tests

Does this PR introduce any user-facing changes?

No