[Kernel] [CC Refactor #3] Add `ConfigurationProvider` and `AbstractCommitCoordinatorBuilder` APIs

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.62k stars 1.71k forks source link

[Kernel] [CC Refactor #3] Add `ConfigurationProvider` and `AbstractCommitCoordinatorBuilder` APIs #3798

Open scottsand-db opened 4 weeks ago

scottsand-db commented 4 weeks ago

Which Delta project/connector is this regarding?

[ ] Spark
[ ] Standalone
[ ] Flink
[X] Kernel
[ ] Other (fill in here)

Description

Adds new ConfigurationProvider and AbstractCommitCoordinatorBuilder APIs

How was this patch tested?

Tests are added in CC Refactor PR 5: https://github.com/delta-io/delta/pull/3821

Does this PR introduce any user-facing changes?

Yes see the above

scottsand-db commented 2 weeks ago

@LukasRupprecht and @sumeet-db -- is it every possible or intended/desired that AbstractMetadata and AbstractProtocol would be null? or are these required params? I see null being passed in in some existing tests today ...

https://github.com/delta-io/delta/blob/6ae4b62845ed579bb5a19f4646831c4ee2931c02/kernel/kernel-api/src/main/java/io/delta/kernel/coordinatedcommits/CommitCoordinatorClient.java#L83

LukasRupprecht commented 2 weeks ago

@LukasRupprecht and @sumeet-db -- is it every possible or intended/desired that AbstractMetadata and AbstractProtocol would be null? or are these required params? I see null being passed in in some existing tests today ...

They shouldn't be null as client implementations may rely on it (test code is an exception if we can be sure that the underlying client doesn't require it).