search

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Apache License 2.0
2.81k stars 1.38k forks source link

Please allow for namespace selector for sparkJobNamespace similar to webhookNamespaceSelector #960

Closed AceHack closed 4 years ago

AceHack commented 4 years ago

It seems weird that there are two ways to specify spark job namespaces, it would be nice to unify this. I would prefer either a list of namespaces or a selector so we can have jobs in more than one namespace.

liyinan926 commented 4 years ago

It's tricky if not impossible to filter based on labels on the namespaces. https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/952 is close to what you ask by providing an option to specify a label selector on the custom resources the operator watches.

AliGouta commented 4 years ago

sparkJobNamespace is not similar to webhookNamespaceSelector. sparkJobNamespace can take all or one single namespace but not a list. However, webhookNamespaceSelector can match labels attached to namespaces. What we proposed in #952 is to further filter on labels attached to pods and custom resources (not to namespaces. Not sure it is doable or if it exists another way to do otherwise). The combination of the two filtering mechanisms allows to run multiple instances of the operator (tested it several times and it works pretty fine. I have 4 instances with ha enabled within the same cluster). @liyinan926 do you think we can merge the PR or shall we dig further ?

AceHack commented 4 years ago

952 would be a great addition and likely meet my needs.

liyinan926 commented 4 years ago

Cool, good to know. Will take a final pass over the PR and merge it.

liyinan926 commented 4 years ago

952 has been merged. Closing this.