delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.56k stars 1.7k forks source link

[Feature Request][Spark] Support Spark 4 + Java 21 #3281

Closed jimsimon-wk closed 4 months ago

jimsimon-wk commented 4 months ago

Feature request

Which Delta project/connector is this regarding?

Overview

Are there plans to upgrade the Spark connector to use Spark 4.x?

Motivation

Further details

It looks like Spark 4.x is also dropping Scala 2.12 in favor of 2.13. Is the Spark connector compatible with Scala 2.13 today?

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

jimsimon-wk commented 4 months ago

🤔 Nevermind, I think I found the plan details! https://github.com/delta-io/delta/releases/tag/v4.0.0rc1