nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

Make finding cluster nodes more reliable in the face of launch failures #362

Closed nchammas closed 9 months ago

nchammas commented 9 months ago

As suggested in #339, this PR updates Flintrock so that it tags launched nodes as part of the launch itself, rather than in a separate call post-launch. This means that we can always rely on tags to find cluster nodes, even when a launch operation fails midway.

This PR also:

I tested this PR by launching and destroying clusters, and interrupting cluster launches at various points to test Flintrock's ability to find the half-baked nodes.

Fixes #183. Fixes #339. Related to #179.