search

stackabletech / spark-k8s-operator

Operator for Apache Spark-on-Kubernetes for Stackable Data Platform
https://stackable.tech
Other
47 stars 2 forks source link

Research if the spark-k8s operator can be used by external tools that only support spark-submit #338

Closed maltesander closed 1 month ago

maltesander commented 6 months ago

Products like Talend only support creating spark-submits. Is there a way how to support simple spark-submits without deploying custom resources? Could be a standby pod receiving spark-submits and creating the actual CR out of that?

This is pretty much against the design of the spark-k8s operator and rather something for archived spark-operator...

razvan commented 6 months ago

spark-submit can work with K8S directly (no need for any operator). Talend just needs to provide the necessary settings (most importantly the K8S crdentials). Of course, in such cases, the experience is crude and the application lifecycle needs to be managed by Talend.

fhennig commented 1 month ago

doesn't make sense to use our operator with spark-submit.