Integrate Spark Streaming with Ray/Ray Datasets

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.

Apache License 2.0

303 stars 66 forks source link

Open carsonwang opened 2 years ago

carsonwang commented 2 years ago

The requirements would be to take a Spark Streaming job and output to a DatasetPipeline that can be used to read the data for training.

DimitarSirakov commented 7 months ago

Is there any progress for this feature ?

kira-lin commented 7 months ago

Hi @DimitarSirakov Unfortunately, due to not enough bandwidth, we are not developing new features for this project now.

DimitarSirakov commented 7 months ago

Is it possible to release the pyspark version lift @kira-lin