search

constellation-rs / constellation

Distributed programming for Rust.
Apache License 2.0
563 stars 24 forks source link

managed clusters #33

Open yvern opened 5 years ago

yvern commented 5 years ago

When dealing with a consolidated distributed computation framework, such as Spark, we could deploy it standalone, as the cluster manager itself.

However, there are also options of "managers" as backends, like Yarn, Mesos or Kubernetes, where they serve as resource providers to Spark.

From what I could understand, Constellation aims to be able to be deployed directly to a group of machines connected by network.

Would it also work on cluster managers? One great advantage of this would be the elasticity feature from cloud providers, or healthchecks/healing from the cluster manager itself.

I understand targeting bare metal could harness more performance by escaping management overhead, but the practicality of having it run on a managed environment could be a nice option to some of us.

Anyway, congrats on the brilliant work!

alecmocatta commented 5 years ago

Thank you @yvendruscolo!

You're spot on and I would love to see for example Kubernetes usable as a backend with Constellation. I think networking might be a little tricky but I'll look into it and get back to you!