search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Spark on Kubernetes - basic scheduler backend [WIP] #492

Closed foxish closed 7 years ago

foxish commented 7 years ago

Stripped out a lot of extraneous things, to create this. Our first PR upstream will likely be similar to this. (note that it is created against the master branch which is up-to-date.) A couple of unit test failures the unit tests for scheduler backend - could be something that changed upstream after 2.2, needs a closer look.

Following PRs will have:

The unnecessary constants and config still needs to be stripped out. Getting this out there to 1) serve as the framework for our first PR, 2) get some insight on the unit test failures. cc @mccheah @ash211 @erikerlandson @felixcheung @rxin

foxish commented 7 years ago

The build fails due to lint-java errors, but the lint errors appear to be an issue on apache/spark:master as well.

ash211 commented 7 years ago

Yeah, for some reason they don't run ./dev/lint-java upstream. I asked about this at https://issues.apache.org/jira/browse/SPARK-21875 but Sean said "Historical reasons" :/

foxish commented 7 years ago

Hmm, I see. Thanks @ash211 for the pointer. Closing this PR in favor of https://github.com/apache-spark-on-k8s/spark/pull/498, because github messed up the diff here.