apache / sedona

A cluster computing framework for processing large-scale geospatial data
https://sedona.apache.org/
Apache License 2.0
1.96k stars 692 forks source link

[SEDONA-668] Drop the support of Spark 3.0, 3.1, 3.2 #1653

Closed jiayuasu closed 3 weeks ago

jiayuasu commented 4 weeks ago

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

Apache Spark has officially announced the EOL of Spark 3.0, 3.1, and 3.2 long time ago. It has been a huge overhead for us to support Sedona on all these versions. We will drop the support of Spark 3.0, 3.1, and 3.2

Here is a summary of current Spark versions on major Spark hosting services

Spark 3.0 Spark 3.1 Spark 3.2 Spark 3.3
Databricks EOL Dec 19, 2024 Mar 18, 2025 Mar 1, 2026
AWS Glue EOL Unknown, Glue 3.0 N/A Unknown, Glue 4.0
AWS EMR July 24, 2025 July 24, 2025 January 25, 2026 January 25, 2026
Microsoft Fabric EOL EOL EOL Unknown
Microsoft Synapse EOL EOL EOL 3/31/2025
Google Dataproc EOL EOL N/A 2024/12/31

This PR did the following:

The following functions will be affected on Spark 3.0, 3.1, 3.2

How was this patch tested?

Did this PR include necessary documentation updates?

jiayuasu commented 4 weeks ago

@jbampton Isort does not throw error on my local machine. It turns out isort often gives inconsistent behavior. I think we should remove it from pre-commit. https://github.com/PyCQA/isort/issues/1889

jbampton commented 3 weeks ago

@jbampton Isort does not throw error on my local machine. It turns out isort often gives inconsistent behavior. I think we should remove it from pre-commit. PyCQA/isort#1889

Just saw this message @jiayuasu

I had a house mate move out so I was busy for about a week.

Thanks.