search

oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
317 stars 70 forks source link

Is it possible for raydp to utilize a Spark session created by Apache Livy? #394

Open Taurus-Le opened 11 months ago

Taurus-Le commented 11 months ago

Hi there, I'm just wondering if it's possible for raydp to utilize a spark session created by Apache Livy? Livy can provide a long running spark session, which helps to reduce the overhead of creating spark session repeatedly.

kira-lin commented 11 months ago

Have you tried it? You can look into python/raydp/context.py to see how we create the session now, and how you can modify it to use Livy.