search

snowplow / dataflow-runner

Run templatable playbooks of Hadoop/Spark/et al jobs on Amazon EMR
http://snowplowanalytics.com
19 stars 8 forks source link

Support temporary cluster-id file for transient cluster #35

Open chuwy opened 6 years ago

chuwy commented 6 years ago

Instead of writing down id of created cluster - we can do following:

  1. After up command create a .cluster-id file containing obviously cluster id in current directory
  2. If --cluster-id was not specified for run - fetch in from the .cluster-id file
  3. Remove the .cluster-id after down
alexanderdean commented 6 years ago

Or potentially in Consul