Watch of CREATE_COMPLETE stack polls forever

jpb commented 7 years ago

$ iidy --profile data --region us-east-1 watch-stack verify-data-cluster-slow-test

Stack Details:
 Name:                    verify-data-cluster-slow-test
 Status                   CREATE_COMPLETE
 Capabilities:            CAPABILITY_IAM,CAPABILITY_NAMED_IAM
 Service Role:            None
 Tags:                    owner=dataops, environment=test, project=ub-data-infrastructure/cluster
 DisableRollback:         false
 Creation Time:           Thu Aug 10 2017 21:21:03
 Last Update Time:        Thu Aug 10 2017 21:21:03
 Timeout In Minutes:      None
 NotificationARNs:        None
 ARN:                     arn:aws:cloudformation:us-east-1:514588561872:stack/verify-data-cluster-slow-test/d0a09c60-7e11-11e7-8c6a-503acac41efd
 Console URL:             https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aus-east-1%3A514588561872%3Astack%2Fverify-data-cluster-slow-test%2Fd0a09c60-7e11-11e7-8c6a-503acac41efd

Previous Stack Events (max 10):
 Thu Aug 10 2017 21:21:11 CREATE_COMPLETE                     AWS::EC2::SecurityGroup                  EC2InstanceSecurityGroup
 Thu Aug 10 2017 21:21:14 CREATE_IN_PROGRESS                  AWS::EC2::Instance                       EC2Instance
 Thu Aug 10 2017 21:21:16 CREATE_IN_PROGRESS                  AWS::EC2::Instance                       EC2Instance
 Thu Aug 10 2017 21:21:26 CREATE_COMPLETE                     AWS::EC2::Volume                         Volume
 Thu Aug 10 2017 21:21:31 CREATE_COMPLETE                     AWS::S3::Bucket                          S3Bucket
 Thu Aug 10 2017 21:21:34 CREATE_IN_PROGRESS                  AWS::IAM::Role                           IAMRole
 Thu Aug 10 2017 21:21:35 CREATE_IN_PROGRESS                  AWS::IAM::Role                           IAMRole
 Thu Aug 10 2017 21:21:49 CREATE_COMPLETE                     AWS::IAM::Role                           IAMRole
 Thu Aug 10 2017 21:21:52 CREATE_COMPLETE                     AWS::EC2::Instance                       EC2Instance
 Thu Aug 10 2017 21:21:56 CREATE_COMPLETE                     AWS::CloudFormation::Stack               verify-data-cluster-slow-test
 10 of 17 total events shown

Live Stack Events (2s poll):
‚¢Ç‚†Ä 58 seconds elapsed

tavisrudd commented 7 years ago

The semantics of what it should do in the case aren't obvious. I think we should inspect the LastUpdatedTime and if it's within the last 5 minutes and the stack is now in a terminal state, assume the operation has finished and exit. Otherwise, we could assume that the terminal state is from a previous operation and the new one is about to begin. We could then optionally set an idel timeout before continuing to watch. Thoughts?

jpb commented 7 years ago

To clarify, you're suggesting:

if the initial state of the stack is CREATE_COMPLETE (or similar), and no new events come in for some time, then stop polling
if the initial state of the stack is CREATE_COMPLETE (or similar), and the stack updated time is more than 5 minutes ago, then don't poll at all

I guess both cases could be collapsed into

if the initial state of the stack is CREATE_COMPLETE (or similar), then poll until stack updated time + 5 minutes, if no new events are received, then stop polling

tavisrudd commented 7 years ago

That works. The timeout could be a cli option with a default of 5.

tavisrudd commented 7 years ago

Fixed with --inactivity-timeout option on watch-stack.

unbounce / iidy

Watch of CREATE_COMPLETE stack polls forever #4