Open InzamamAnwar opened 1 month ago
Hi,
Do the Python and Spark versions of the "Jupyterhub Serve" and Image "spark:3.5.0" match?
Although written in Korean, I recently tested the "spark:3.5.0" image with Jupyter Notebook. I hope this helps.
[url] https://github.com/ha2hi/spark-study/tree/main/spark-on-k8s/Jupyter-Notebook https://github.com/ha2hi/spark-study/tree/main/spark-on-k8s/Jupyter-Hub
Please describe your question here
How to integrate Jupyterhub with Spark Operator? Tried to integrate it by installing Jupyterhub and Spark-operator but it's not working
Jupyterhub and Spark operator are running in the same namespace. Attached spark-operator service account to jupyterhub so that it can communicate with Kubernetes APIs. Creating a SparkSession through the following code,
The executors are being creating and killed right after they are created. Cannot see driver pod anywhere. The error we get with the above is given below;
Can anyone please help in this regard?
Provide a link to the example/module related to the question
Additional context