Spark with native support for Nomad?

hashicorp / nomad-spark

DEPRECATED: Apache Spark with native support for Nomad as a scheduler

44 stars 16 forks source link

Spark with native support for Nomad? #8

Closed Hushpar closed 4 years ago

Hushpar commented 6 years ago

Spark 2.3.0 now have native K8S support http://blog.kubernetes.io/2018/03/apache-spark-23-with-native-kubernetes.html Any plans to do the same with Nomad?

Lwelch45 commented 6 years ago

There was a thread a while back around adding native support for nomad. It seemed to not go anywhere, due in partly to politics but also because they want to move to a model where anyone can add support for their resource management system. I don't have the ticket on hand but it was closed a while ago. Given the pace of the hasicorp team I expect that they will add it soon.

rkettelerij commented 6 years ago

@Lwelch45 I think you're referring to this ticket: https://issues.apache.org/jira/browse/SPARK-19700

cgbaker commented 6 years ago

The hope is to get this into upstream Spark or using a plugin system like in the link above; we will keep this ticket up-to-date regarding our progress in that.

OneCricketeer commented 4 years ago

Duplicates https://github.com/apache/spark/pull/18209

cgbaker commented 4 years ago

We have decided to focus our ecosystem efforts on other projects; this project is now deprecated. I have updated the README with information to this effect.