search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.18k stars 1.62k forks source link

[Spark] Refactors InMemoryCommitOwnerClient suite into a generic suite that can be implemented by other CommitOwnerClient implementations #3075

Closed LukasRupprecht closed 1 month ago

LukasRupprecht commented 1 month ago

Which Delta project/connector is this regarding?

Description

This PR refactors the existing InMemoryCommitOwnerClientSuite into a generic suite, CommitOwnerClientImplSuiteBase. This base suite can be extended by CommitOwnerClient implementations to run the basic set of tests provided by the base suite. This is to make it easier for CommitOwnerClient implementations to get test coverage in the future.

It also adds an implementation of the suite for the InMemoryCommitOwnerClient with two different backfilling batch sizes (1 and 5).

How was this patch tested?

Ensured that the newly introduced suites pass.

Does this PR introduce any user-facing changes?

No

sumeet-db commented 1 month ago

This should really help any new CommitOwners easily validate the basic functionality!