search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Set Parameter kubernetes-namespace via --conf #598

Closed mabunixda closed 6 years ago

mabunixda commented 6 years ago

Hi,

i am currently working on using spark on k8s with some spark worflow services ( livy + hue, azkaban, ... ) This works really nice but i just ran into one problem: The kubernetes-namespace cannot be set within those services because it cannot be set via --conf parameter setting. Because it is implemented via --kubernetes-namespace all those tools must be adapted to support the parameter.

In my opinion it would be much easier to adapt the setting within spark-k8s to use the --conf setting parameter and be compatible with the other parameters than rewriting/adapting all other systems on the layers above spark :(

Martin

liyinan926 commented 6 years ago

Under the hood, kubernetes-namespace is translated to spark.kubernetes.namespace. So alternatively you can use the --conf option to set spark.kubernetes.namespace directly. In upstream Spark 2.3, the option kubernetes-namespace has been removed so using spark.kubernetes.namespace is the only way available.

mabunixda commented 6 years ago

Oh damn nice :) Was that written somewhere within the docu?

When spark 2.3 is released, will apache-spark-on-k8s be closed? Where will furhter dev work happen?

Thanks!

liyinan926 commented 6 years ago

https://github.com/apache-spark-on-k8s/userdocs/blob/master/src/jekyll/running-on-kubernetes.md#spark-properties.

No, we won't close this repo as there are still features yet to be upstreamed. Development will likely continue to happen in this repo.

mabunixda commented 6 years ago

ok thanks :)