delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.22k stars 1.62k forks source link

[Feature Request] Consolidate optimizer suite logic across compaction and z-order #3249

Open Kimahriman opened 2 weeks ago

Kimahriman commented 2 weeks ago

Feature request

Which Delta project/connector is this regarding?

Overview

Follow up from https://github.com/delta-io/delta/pull/3089#discussion_r1632562644. Try to consolidate some of the common code, particularly the optimizeWithBatching helper, into a common function or base class that can be shared across compation and z-order suites.

Motivation

Code deduplication/consolidation.

Further details

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?