search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Spark on Kubernetes - basic scheduler backend #498

Closed foxish closed 6 years ago

foxish commented 7 years ago

Continuing https://github.com/apache-spark-on-k8s/spark/pull/492

Stripped out a lot of extraneous things, to create this. Our first PR upstream will likely be similar to this. (note that it is created against the master branch which is up-to-date.)

Following PRs will have:

TODO before we can retarget this to apache/spark:master:

cc @ash211 @mccheah @apache-spark-on-k8s/contributors

mccheah commented 7 years ago

Checkstyle on Java is what's causing the build failures.

foxish commented 7 years ago

Addressed comments. Thanks @mccheah and @kimoonkim!

foxish commented 7 years ago

Unit tests look good. There's not much else that can be tested at this point. If there are no further comments, I'll turn this into a PR against upstream tomorrow at 5pm.

mccheah commented 7 years ago

@felixcheung private[spark] has been canonically used everywhere but I don't know if there is a set convention to scope these down further.

mccheah commented 7 years ago

@foxish cherry-picking #491 for this too, correct?

foxish commented 7 years ago

The relevant parts of #491 are already covered in this PR.

mccheah commented 7 years ago

...Once the build is fixed, of course =)

felixcheung commented 7 years ago

It should, for example sql has private[sql] and mesos has private[mesos]

https://github.com/apache/spark/blob/master/resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala

foxish commented 7 years ago

This is being reviewed in https://github.com/apache/spark/pull/19468

felixcheung commented 6 years ago

could you check the Integration Test? it seems to have failed to build [ERROR] Could not find the selected project in the reactor: resource-managers/kubernetes/integration-tests @

mccheah commented 6 years ago

@felixcheung we shouldn't be running integration tests when we submit upstream. Also, shouldn't we instead be tracking https://github.com/apache/spark/pull/19717 and thus this PR be closed?

foxish commented 6 years ago

This PR is tracking the changes in the upstream one. It's the same branch used for both PRs, and is simply for tracking.

On Nov 28, 2017 10:32 AM, "mccheah" notifications@github.com wrote:

@felixcheung https://github.com/felixcheung we shouldn't be running integration tests when we submit upstream. Also, shouldn't we instead be tracking apache#19717 https://github.com/apache/spark/pull/19717 and thus this PR be closed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache-spark-on-k8s/spark/pull/498#issuecomment-347619545, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3U5xdA-jTh_3QQGwd4KGQ-1C15LdWyks5s7FGxgaJpZM4PdiJJ .

liyinan926 commented 6 years ago

Closing this one as the upstream PR has been merged.