Update RayDP to be compatible with Ray 2.5.0

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.

Apache License 2.0

293 stars 66 forks source link

Update RayDP to be compatible with Ray 2.5.0 #354

Open kira-lin opened 1 year ago

kira-lin commented 1 year ago

We cannot support Ray 2.5.0 as Ray Data batcher does not consider block type bytes. Even if we push a fix to Ray now, Ray 2.5.0 won't be updated. After our arrow serialization PR is merged, this problem will be fixed.