InvalidChangeSetStatus: Cannot delete ChangeSet in status CREATE_IN_PROGRESS

[miltador]

I often come up with the mentioned error while provisioning my service. The process then shuts down. I have to relaunch the provisioning process several times then to get it working.

The command I use is: go run main.go provision --s3Bucket $S3_BUCKET --tags staging

Output log:

INFO[0009] Uploading to S3                               Path="/home/miltador/Projects/Go/src/path_to_project/.sparta/project_staging-cftemplate.json"
INFO[0011] Issued CreateChangeSet request                StackName=project-staging
INFO[0025] No changes detected for service               StackName=project-staging
INFO[0025] Invoking rollback functions                   RollbackCount=2
INFO[0025] Deleting S3 object                            URL="https://some-bucket-eu-west-1.amazonaws.com/project-staging/project_staging-cftemplate-da84d441e598ed0b2e754c4717f1615138565a3c.json"
INFO[0025] Deleting S3 object                            URL="https://some-bucket-eu-west-1.amazonaws.com/project-staging%2Fproject_staging-code-98a0481e4fd893a007481e92e45f2add325e1ce1.zip"
ERRO[0025] InvalidChangeSetStatus: Cannot delete ChangeSet in status CREATE_IN_PROGRESS
        status code: 400, request id: 5d86eba7-ef3f-11e6-8037-0dfba0579a7d 
Error: InvalidChangeSetStatus: Cannot delete ChangeSet in status CREATE_IN_PROGRESS
        status code: 400, request id: 5d86eba7-ef3f-11e6-8037-0dfba0579a7d

[mweagle]

Hi @miltador , I can think of a couple of explanations for this. I have a few more questions to help diagnose:

1. What version of Sparta are you using?
2. If you enable --debug log do you see any more informative log messages?
3. Is there any chance you're running the same deployment on another machine (eg, CI server)?
4. When this failure is happening, what do you see in the AWS Console at CloudFormation -> project-staging -> Change Sets (tab)?

[miltador]

Hi, @mweagle.

1. I am using latest Sparta master (it outputs v0.10.0 in console).
2. I emailed it you.
3. Only one deployment at time.
4. Emailed.

I can also tell that once this error occurs, if I wait for 1-2 minutes and rerun, it then provisions the project for about ~10 minutes to completion which is loooong.

[mweagle]

Thanks for the info @miltador

This looks like a race condition when quickly creating/destroying CloudFormation ChangeSets:

[15:41:38][Step 3/3] time="2017-02-14T15:41:38+01:00" level=info msg="No changes detected for service" StackName=LicensePortalService-staging 

When the empty change set is detected, Sparta attempts to delete the AWS changeset, but as it's still in CREATE_IN_PROGRESS mode, the subsequent delete fails.

When this happens are there local changes that you're trying to push to AWS, or is this more analogous to an ensure operation?

[miltador]

Mostly it's just updating of my functions code, not changing AWS configuration itself or adding new functions.

[mweagle]

Sparta could retry in a polling loop, but that's more a mitigation than a solution. Does this happen all the time? What's your typical workflow in terms of publishing? When this happens, does the Sparta output match what you see in the AWS console?

[miltador]

This happens every day. Currently build server publishes the service to the staging stage on every change in VCS. When it happens I think it matches to what I see in AWS console, and in ChaneSets the change set quickly changes status from yellow to green.

[mweagle]

Hi @miltador - keeping this open for the next Sparta release to see if there's a better way to handle the race condition.

[mweagle]

Hi @miltador - encountered this issue today. Looking to see how it might be mitigated.

[mweagle]

Hi @miltador - finally was able to reproduce this. Resolved in Sparta 0.11.2. Thanks!