Spark Operator: Installation and access via KF Pipelines

[chritter]

Is your feature request related to a problem? Please link issue ticket

We would need to run Spark jobs on AAW as part of a Kubeflow model training pipeline.

Describe the solution you'd like

Being able to run spark jobs as part of a Kubeflow pipeline component: https://www.kubeflow.org/docs/components/pipelines/concepts/component/#component-code

Describe alternatives you've considered

Run a spark job in the CAE/Azure service: https://github.com/kubeflow/pipelines/tree/master/components/azure/azuresynapse/runsparkjob

Additional context

[chritter]

According to @blairdrummond the spark operator is installed so I will need to test them.

[StanHatko]

Since Kubeflow pipelines are being removed on AAW I think this issue can be closed.