Support KeepJobFlowAliveWhenNoSteps Or Auto-termination (after idle) in stepfunction creates EMR cluster.
Use Case
Our team is using Stepfunction EMR (EmrCreateCluster function) to build our ETL pipeline.
While the emr cluster are working very well on creating, add steps and the cluster is terminated properly when stepfunction is finished or failed, it remains "waiting" status when we manually abort the step function or the step function timeout.
Wondering if we can export the KeepJobFlowAliveWhenNoSteps as an optional parameter when create the Cluster, or have an option to configure auto-termination after idle time.
Proposed Solution
No response
Other Information
No response
Acknowledgements
[X] I may be able to implement this feature request
pahud
commented
8 months ago
Describe the feature
Support KeepJobFlowAliveWhenNoSteps Or Auto-termination (after idle) in stepfunction creates EMR cluster.
Use Case
Our team is using Stepfunction EMR (EmrCreateCluster function) to build our ETL pipeline.
While the emr cluster are working very well on creating, add steps and the cluster is terminated properly when stepfunction is finished or failed, it remains "waiting" status when we manually abort the step function or the step function timeout.
Wondering if we can export the KeepJobFlowAliveWhenNoSteps as an optional parameter when create the Cluster, or have an option to configure auto-termination after idle time.
Proposed Solution
No response
Other Information
No response
Acknowledgements
CDK version used
2.14.0
Environment details (OS name and version, etc.)
N/A