Improve transient cluster termination policy

snowplow / dataflow-runner

Run templatable playbooks of Hadoop/Spark/et al jobs on Amazon EMR

http://snowplowanalytics.com

19 stars 8 forks source link

Improve transient cluster termination policy #76

Closed oguzhanunlu closed 2 years ago

oguzhanunlu commented 2 years ago

0.6.0's auto termination uses default policy of AWS SDK which results in ResourceNotReady: exceeded wait attempts for long-running jobs since it has 30 min expiration.

chuwy commented 2 years ago

Hey @oguzhanunlu! Please don't forget to set milestones.