Copilot SVC deployment fails when add-ons contain Serverless transform

[gautam-nutalapati]

Copilot version 1.7.1.

When Using serverless transform AWS::Serverless-2016-10-31 in add-ons template, Service deployment fails when updating ECS task with new docker image without change to add-ons.

Error seen:

✘ Proposing infrastructure changes for stack vault-dev-test-redeploy
✘ deploy service: wait for creation of change set copilot-1c1ba4be-e30f-4fcd-92a6-a4a0f9b7a995 for stack vault-dev-test-redeploy: ResourceNotReady: failed waiting for successful resource state

Further investigation:

WithOut AWS::Serverless-2016-10-31: Add-ons stack change set status is failed with reason The submitted information didn't contain changes. Submit different information to create a change set.. But deployment is successful.

With AWS::Serverless-2016-10-31: Add-ons stack change set status is failed with reason No updates are to be performed. and deployment fails.

Deeper investigation :)

Using custom built copilot with some extra logs, was able to pin point where the issue happens. After change set is created, copilot waits for completion of change set creation and error is thrown. Link Below err is thrown:

UNAVAILABLE Nested change set arn:aws:cloudformation:us-east-1:XXX:changeSet/copilot-d9ff15e1-f0a6-438d-b7fa-c8ba6016276e-AddonsStack-1ALG59ZUAC9UQ/217b070c-c7b4-4f1f-9384-5eb4a011874c was not successfully created: Currently in FAILED. 2021-05-24 16:35:25.436 +0000 UTC

Summary:

If add-ons template containing AWS::Serverless-2016-10-31 has no changes compared to previous deployment, I am not sure why root stack change-set is detecting changes in add-ons stack and looks for its status, which is in Failed state and then whole deployment fails. Where as add-ons template NOT containing AWS::Serverless-2016-10-31, the root stack change-set doesn't see any add-ons stack changes, and thus root stack deployment passes, even though add-ons stack is in failed state.

How to reproduce:

Simply add Transform: AWS::Serverless-2016-10-31 to any add-ons template and try to deploy new docker image via copilot svc deploy -e <ENV> --tag <NewTag>

Sample template I used:

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  svcName:
    Type: 'AWS::SSM::Parameter'
    Properties:
      Description: svcName.
      Name: !Sub /${App}/${Name}/${Env}/svcName
      Type: String
      Value: !Sub ${Name}
      Tags:
        env: !Sub ${Env}
        app: !Sub ${App}
        svc: !Sub ${Name}

This prevents us from deploying new version of service hence a critical blocker.

[huanjani]

Hello @gautam-nutalapati!

This situation reminds me of the issue here and here. Let me know if you think it's the same root problem.

Thanks!

[gautam-nutalapati]

Hi @huanjani thanks for quick response. Yes root cause is same.

I see in 1.7.0, condition strings.Contains(descr.StatusReason, "didn't contain changes" was added to avoid the deployment failure when nested stack didn't have any changes. When we use transform, cloudformation seems to have a diferent stack status reason No updates are to be performed.

[huanjani]

Yes, when this issue first came up a couple of months ago, we contacted the CloudFormation team to propose that this is a bug on their end. This issue seems deeper than Copilot; we will dig into this again to see what we can do.

I'm sorry you are being blocked by this; as a work-around, can you make a nominal change to the add-ons template to generate a change set?

[gautam-nutalapati]

Yes, Making a nominal change to add-ons template is our temporary approach to get things deployed.