nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
637 stars 116 forks source link

Track Java version in cluster manifest #333

Closed nchammas closed 3 years ago

nchammas commented 3 years ago

Oversight on my part. add-slaves should not have or need the ability to specify a Java version. It should source it from the cluster manifest, guaranteeing that all added slaves have the same configuration as the existing cluster.

This matches the behavior of add-slaves in regards to picking the appropriate version of Hadoop or Spark to install on the new nodes.

Potential future improvement: Make Java a "service" like Hadoop and Spark, and reuse the cluster provisioning abstractions already built for them. Will need to build a way to express service dependencies (e.g. Spark depends on Java) so they get installed in the correct order.

Follow-up to #316.