search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
293 stars 66 forks source link

Is it possible for raydp to utilize a Spark session created by Apache Livy? #394

Open Taurus-Le opened 6 months ago

Taurus-Le commented 6 months ago

Hi there, I'm just wondering if it's possible for raydp to utilize a spark session created by Apache Livy? Livy can provide a long running spark session, which helps to reduce the overhead of creating spark session repeatedly.

kira-lin commented 6 months ago

Have you tried it? You can look into python/raydp/context.py to see how we create the session now, and how you can modify it to use Livy.