nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

Fix some "flintrock describe" errors, using a filter on instance-state-name #246

Closed dm-tran closed 6 years ago

dm-tran commented 6 years ago

This PR makes the following change:

I tested this PR by

For each tested command, I ran "flintrock describe" using two standalone Flintrocks (one with the fix and one without).

This PR partially fixes Exception: Could not extract cluster name from instance: when a cluster is destroyed or when slaves are removed (#236): this exception will be raised during the first seconds instead of being raised during 30 seconds / a minute.

This PR also fixes the following error, which is raised when a cluster is launched or when slaves are added:

Traceback (most recent call last):
  File "standalone.py", line 11, in <module>
  File "flintrock/flintrock.py", line 1110, in main
  File "click/core.py", line 716, in __call__
  File "click/core.py", line 696, in main
  File "click/core.py", line 1060, in invoke
  File "click/core.py", line 889, in invoke
  File "click/core.py", line 534, in invoke
  File "click/decorators.py", line 17, in new_func
  File "flintrock/flintrock.py", line 477, in describe
  File "flintrock/ec2.py", line 1053, in get_clusters
  File "flintrock/ec2.py", line 1053, in <listcomp>
  File "flintrock/ec2.py", line 1104, in _compose_cluster 
  File "flintrock/ec2.py", line 1080, in _get_cluster_master_slaves
TypeError: 'NoneType' object is not iterable

Instead of raising this error, flintrock describe will now display :

my-cluster-name:
  state: inconsistent
  node-count: 0
nchammas commented 6 years ago

Thank you for submitting this @dm-tran! Your changes look good to me.

This PR also fixes the following error, which is raised when a cluster is launched or when slaves are added:

To be clear, do you mean this error is raised if you call flintrock describe while another instance of Flintrock is launching or expanding a cluster? Or do you mean that flintrock launch is sometimes throwing this error directly?

dm-tran commented 6 years ago

@nchammas: TypeError: 'NoneType' object is not iterable is raised if I call flintrock describe while another instance of Flintrock is launching or expanding a cluster.

nchammas commented 6 years ago

OK, thanks for clarifying.