[SPARK] Stringify the delta optimize 'auto' operationalParameter in CommitInfo - Githubissues

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.62k stars 1.71k forks source link

[SPARK] Stringify the delta optimize 'auto' operationalParameter in CommitInfo #3889

Open MaicoTimmerman opened 2 days ago

MaicoTimmerman commented 2 days ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

The inCommitTimestamp feature of Delta Kernel API expects the operationalParameters of CommitInfo to be a MapType<StringType, StringType>.

The newly added autoCompact feature added the auto flag to operationalParamaters as a boolean, which is incompatible.

This PR restores that expectation of the inCommitTimestamp feature for the OPTIMIZE operation in the Spark Connector.

Resolves #3888.

How was this patch tested?

I've attached a test-case.

Does this PR introduce any user-facing changes?

We change format of the field commitInfo.operationalParameters.auto field from boolean to string.