search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Secret for shipping local dependencies gets created when no local file is used #556

Closed liyinan926 closed 6 years ago

liyinan926 commented 6 years ago

I noticed that a secret named with suffix -submitted-files gets created even when no local file dependency is used. Below is part of the spark-submit command I used. Both the jar files specified through --jars and the main application jar are remote dependencies. My understanding is such a secret should only be created when no RSS gets specified and there's local files to ship to the driver/executors.

bin/spark-submit \
...
--jars http://central.maven.org/maven2/com/databricks/spark-avro_2.11/4.0.0/spark-avro_2.11-4.0.0.jar,http://central.maven.org/maven2/com/google/guava/guava/19.0/guava-19.0.jar
...
gs://spark-k8s/jars/app.jar

Below is the result of running kubectl get secrets. 

ynli-macbookpro:~ ynli$ kubectl get secrets
NAME                                                     TYPE                                  DATA      AGE
default-token-9ddgs                                      kubernetes.io/service-account-token   3         5d
spark-09e8db9d3478410ab023d7907ea5ef6f-submitted-files   Opaque                                0         12m

@foxish @mccheah