search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

Version 2.1.0 is not backward compatible with clusters created with version 2.0.0 #374

Closed sfcoy closed 3 months ago

sfcoy commented 3 months ago

This is an FYI. I don't expect it to be fixed.

It fails because the older clusters do not have any ec2 tags named "flintrock-name".

I fixed our clusters using the AWS CLI:

aws ec2 create-tags --resources i-abc134... i-def456... --tags Key=flintrock-name,Value=foo

where the cluster has a master node named foo-master and the slaves are all foo-slave.

nchammas commented 3 months ago

Thanks for sharing. Yes, this is a deliberate design decision:

For starters, Flintrock provides no guarantee that clusters launched with one version of Flintrock can be managed by another version of Flintrock, and no considerations are made for any long-term use cases.

This choice is mostly in service of making it easier to maintain Flintrock over a long period of time. I'm glad the fix in your case was straightforward.