search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
293 stars 66 forks source link

Add shim layer for Spark 3.4.0 and 3.2.4 #342

Closed kira-lin closed 1 year ago

kira-lin commented 1 year ago

implement shim for spark 3.4.0 add support for spark 3.2.4

closes #338

carsonwang commented 1 year ago

Thanks!

pang-wu commented 9 months ago

@carsonwang @kira-lin This doesn't seems to work on Spark 3.4.x, we tried out raydp 1.6.0 with Spark 3.4.1 on Ray 2.6.1 We see actors coming up but Spark executors are not registered..