Can run Spark in mode client when using Livy

night2201 commented 4 months ago

Hi I am testing Jupyter Notebook with spark on k8s in mode cluster. However, I faced with some problems:

I can't copy spark jar files from local notebook to spark cluster in k8s (I find that jar files are copied from driver to executor) so I met error: not found file.
And when connecting to spark through livy. The working directory on the notebook was changed to lied on spark driver

So, can run spark in mode client through Livy? Now, I realize that in client mode will resolve the above problem

askhatri commented 4 months ago

Hi @night2201 , Yes, it's definitely possible to achieve spark in mode client. This approach is suitable for smaller Spark applications or when you want to leverage the resources of an existing Kubernetes cluster without managing a dedicated Spark environment.

In order to use Jupyter Notebook with spark on k8s in mode cluster, you can add K8s deployment, service in helm chart and test it. In order to solve spark jar files copy problem from local notebook to spark cluster in k8s, you can use PersistentVolumeClaim from "livycluster/templates/pvc.yaml" and share the jar files across Jupyter Notebook and Spark pods.

You can fork this repo and add your helm chart for Jupyter Notebook so that I can debug and try to help you further on this.

night2201 commented 4 months ago

Thanks @askhatri for replying, I will check

night2201 commented 4 months ago

Hi @askhatri, I have tested with jupter notebook. But I faced an error I enable hive context following this link but it's not working for me

https://github.com/jupyter-incubator/sparkmagic/issues/305

I use the same image in helm. Currently, I don't have solution for this problem. Do you have any idea for it?

askhatri commented 4 months ago

Hi @night2201 , No. I am not sure about this error.

askhatri / livycluster

Can run Spark in mode client when using Livy #1