search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Spark Submit changes and test #542

Closed foxish closed 6 years ago

foxish commented 6 years ago

cc @mccheah @liyinan926 @apache-spark-on-k8s/contributors

ifilonenko commented 6 years ago

Are we missing testing logic for app resources or is that going to be a follow up PR?

foxish commented 6 years ago

@ifilonenko Added the check for the jar that's submitted. Did you mean something else? The implementation details wrt resources should be covered under org/apache/spark/deploy/k8s/submit/ClientSuite.scala and submission step tests I think.

ifilonenko commented 6 years ago

@foxish that was part of it, yes. But coverage in regards to this: https://github.com/apache-spark-on-k8s/spark/blob/0add46a0240d74fb4f746e100ee9a7d6d29d87bd/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L636-L653

foxish commented 6 years ago

Ah! I see.. I think those should be separate unit tests in a subsequent PR. I was looking at this from the perspective of the upstreaming, and initially, we're not going to have python/R support.

foxish commented 6 years ago

rerun integration test please

foxish commented 6 years ago

rerun unit tests please

foxish commented 6 years ago

rerun integration test please

foxish commented 6 years ago

rerun integration test please