[SEDONA-663] Support spark connect python api - Githubissues

apache / sedona

A cluster computing framework for processing large-scale geospatial data

https://sedona.apache.org/

Apache License 2.0

1.96k stars 692 forks source link

[SEDONA-663] Support spark connect python api #1639

Closed sebbegg closed 1 month ago

sebbegg commented 1 month ago

Did you read the Contributor Guide?

Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a JIRA ticket?

Yes: https://issues.apache.org/jira/browse/SEDONA-663

What changes were proposed in this PR?

This PR enables the python dataframe api to be used in spark-connect sessions. See https://github.com/apache/sedona/issues/1619 for the current status.

How was this patch tested?

Ran tests/sql/test_dataframe_api.py locally against a (dockerized) spark connect session using sedona 1.6.1.
Successfully ran the python build workflow for spark 3.5

Did this PR include necessary documentation updates?

No, this PR does not affect any public API so no need to change the documentation.

_{Sebastian Eckweiler (sebastian.eckweiler@mercedes-benz.com), Mercedes-Benz AG on behalf of MBition GmbH (More Info)}

sebbegg commented 1 month ago

Switched to use UnresolvedFunction instead of CallFunction which is only available for spark >= 3.5. UnresolvedFunction seems "stable" in that it exists in spark >= 3.4, including the 4.0 preview releases.