search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.32k stars 445 forks source link

Jupyterhub + Sparkmagic/levy + Spark on K8s? #575

Open neerawrt opened 4 years ago

neerawrt commented 4 years ago

Does anyone have Jupyterhub + Sparkmagic/levy + Spark running on Kubernetes at scale (in production)?

How stable is it?

itamarst commented 4 years ago

I have no personal experience, but apparently Amazon EMR does Sparkmagic: https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/

ogidogi commented 4 years ago

Rather fine, but have some minor issues. Already 3 prod deployments. Based on this version: https://github.com/apache/incubator-livy/pull/167 Plus some small patches.

itamarst commented 4 years ago

@ogidogi if you have any useful patches to Sparkmagic, could you submit them as PRs? Thanks!

jahstreet commented 4 years ago

Feel free to refer the Helm setup ready for use: https://github.com/jahstreet/spark-on-kubernetes-helm . I'm opened for questions and feedback on usage. Can assist you with deployment.

ogidogi commented 4 years ago

Also some information can be found here: https://github.com/apache/incubator-livy/pull/249

PedroRossi commented 4 years ago

@neerawrt I am currently using this exact architecture and the only problem I have found is that sparkmagic suport for jupyterlab has some bugs on the progress bar and other visual issues using the scala kernel

wjxiz1992 commented 1 year ago

Hi @PedroRossi , is there any luck for us to see the sharing of how this architecture is set up on k8s?

devstein commented 1 year ago

Hi @wjxiz1992 there a some resources online for Sparks on K8s + Apache Livy. I know Viaduct has some open source repos for their docker images: https://github.com/viaduct-ai

Wh1isper commented 1 year ago

Hey guys, I've developed a module for deploying the latest 3.4.0 server-client mode on k8s and support config PySpark Session for direct connections!

https://github.com/Wh1isper/sparglim#spark-connect-server-on-k8s

Ping @devstein @wjxiz1992 @PedroRossi @jahstreet @ogidogi @itamarst @ogidogi @neerawrt