update to use Spark 3.3

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.

Apache License 2.0

315 stars 69 forks source link

update to use Spark 3.3 #253

Closed kira-lin closed 2 years ago

kira-lin commented 2 years ago

better to use spark shim in 3.3

carsonwang commented 2 years ago

We can drop Python 3.6. PySpark 3.3 requires Python 3.7+.

kira-lin commented 2 years ago

ok, but we'll need a shim layer to add support for 3.3. I've told @KepingYan to look into it. I 'll drop python 3.6 here, and it seems like raydp can work with ray 1.13.

carsonwang commented 2 years ago

LGTM