Create plain Jupyter notebook ROCK based on Spark ROCK and test

[misohu]

Goal is to try to submit spark jobs inside Kubeflow cluster by using the work from the spark team (https://github.com/canonical/charmed-spark-rock). To achieve this we need to create custom ROCK based on the ROCK of spark team with just simple jupyter lab on top of it.

DoD:

• Have plain jupyter notebook (jupyter lab) with spark inside
• Test spark job creation and monitoring inside cluster from that notebook.

[misohu]

Ideally when testing this ROCK in Kubeflow cluster we want to use service account which comes with the kubeflow profile to submit the spark job. The job is just a pod.

[kimwnasptd]

Now that the DP team has created their own ROCK https://github.com/canonical/charmed-spark-rock/pull/43 with JupyterLab and PySpark we'll need to understand how could we re-use part of it

Considerations that come to mind:

1. Versions fo python and java
2. They define the JupyterLab command as an env var. But we need to dynamically set the prefix of the notebook https://github.com/canonical/kubeflow-rocks/issues/54
3. We'll need to update the, if we'd use the same, command to specify the namespace of spark. Shouldn't it be the same of the notebook's?

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5213.

This message was autogenerated