[ecs] EfsVolumeConfiguration property gets removed from Green TaskDefinition during CfnCodeDeployBlueGreenHook transform

[zechariahks]

I am using CDK to implement ECS Blue Green deployment using BlueGreen deployment hook. For this, I am using the cfn core construct CfnCodeDeployBlueGreenHook as the high level construct is not yet available. Everything works fine during the creation of the stack. cdk deploy creates the stack with all Blue resources (Blue TaskDefinition, Blue TaskSet etc.,) successfully.

However, If I perform an update on the stack by modifying the image: property in CDK code, during the stack update, it is removing the EfsVolumeConfiguration property from the Volumes in the BlueTaskDefinition and also the Green TaskDefinition resources. This results in an unexpected behavior where BlueTaskDefinition and BlueTaskset resources are re-created along with Green resources. As per the Blue Green workflow, it should only create Green resources and remove Blue resources.

The issue seems to be with the camel casing of EfsVolumeConfiguration property. With camel case, only creation of the stack works correctly but stack update leads to modifying the Volumes section of the task definition when there are no changes submitted for Volumes.

If we modify EfsVolumeConfiguration property to EFSVolumeConfiguration property in the synthesized template or modify it in the CDK app using escape hatches, then the stack creation and update works as expected.

Reproduction Steps

1. Create a CDK app using the code here

Example command: cdk deploy --parameters VPC=vpc-41effb26 --parameters Subnet1=subnet-14ecff4f --parameters Subnet2=subnet-af4087c9

2. Blue TaskDefinition etc., gets created successfully in the stack. You can validate that efsVolumeConfiguration properly configured in the task definition

aws ecs describe-task-definition --task-definition "ecs-efs:1" --query taskDefinition.volumes

[
    {
        "name": "myefs",
        "efsVolumeConfiguration": {
            "fileSystemId": "fs-cfad773a",
            "rootDirectory": "/",
            "transitEncryption": "ENABLED"
        }
    }
]

3. In CDK code, update the image: property as below so that BlueGreen hook triggers Green resources.

 image: ecs.ContainerImage.fromRegistry("nginxdemos/hello:latest"),

4. Run below commands to create the synthesized template.

npm run build

cdk synth > template.yaml

Here if we run cdk deploy, stack will replace (instead of remove) BlueTaskDefinition, BlueTaskSet and create GreenTaskDefinition, GreenTaskSet. Due to this, At the end of the deployment, ECS Service will end up with three task sets and stack will get suck in "UPDATE_COMPLETE_CLEANUP_IN_PROGRESS" state for a long time.

For the purpose of reproducing the actual issue, let's create only the change set with synthesized template.

aws cloudformation create-change-set --stack-name EcsFargateEfsStack --change-set-name bluegreen --template-body file://template.yaml --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM --parameters ParameterKey=VPC,UsePreviousValue=true ParameterKey=Subnet1,UsePreviousValue=true ParameterKey=Subnet2,UsePreviousValue=true

If you check the change set JSON Changes in the CloudFormation console, it shows that it is modifying "Volumes" when we are not changing anything related to Volumes in the task definition.

{
    "resourceChange": {
      "logicalResourceId": "BlueTaskDefinitionD77B7F7A",
      "action": "Modify",
      "physicalResourceId": "arn:aws:ecs:us-west-1:413988070266:task-definition/ecs-efs:1",
      "resourceType": "AWS::ECS::TaskDefinition",
      "replacement": "True",
      "moduleInfo": null,
      "details": [
        {
          "target": {
            "name": "Volumes",
            "requiresRecreation": "Always",
            "attribute": "Properties"
          },
          "causingEntity": null,
          "evaluation": "Static",
          "changeSource": "DirectModification"
        }
      ],
      "scope": [
        "Properties"
      ]
    },
    "type": "Resource"
  },
  {
    "resourceChange": {
      "logicalResourceId": "BlueTaskDefinitionD77B7F7A",
      "action": "Remove",
      "physicalResourceId": "arn:aws:ecs:us-west-1:413988070266:task-definition/ecs-efs:1",
      "resourceType": "AWS::ECS::TaskDefinition",
      "replacement": null,
      "moduleInfo": null,
      "details": [],
      "scope": []
    },
    "type": "Resource"
  },

If execute the change set, it will replace Blue resources and create Green resources. If you check the green task definition that got created it will not contain the efsVolumeConfiguration.

aws ecs describe-task-definition --task-definition "ecs-efs:4" --query taskDefinition.volumes

[
    {
        "name": "myefs",
        "host": {}
    }
]

During the transform, CodeDeploy BlueGreen hook might be using the EFSVolumeConfiguration in the schema definition and resulting in removal of this property during the update operation.

If we modify the Volumes section as below, it works fine. EFSVolumeConfiguration and FilesystemId need to be in proper case. Please note that you might need to delete the stack which was created previously and re-create and update using the modified template.

Volumes:
        - Name: myefs
          EFSVolumeConfiguration:
            TransitEncryption: ENABLED
            FilesystemId:
              Ref: efs6C17982A

What did you expect to happen?

It should not delete the efsVolumeConfiguration property in the Task Definitions. It should delete the blue resources and create the green resources.

What actually happened?

As the Volumes property getting modified during the transform, it is re-creating blue resources and creating green resources. And also, it is removing the efsVolumeConfiguration from Volumes section in the task definitions.

Environment

• CDK CLI Version : 1.73.0 (build eb6f3a9)
• Framework Version: 1.73.0
• Node.js Version: v14.15.0
• OS : Mac OS X 10.15.7
• Language (Version): TypeScript Version 4.0.5

Other

Currently, there are two workarounds. Requesting if EfsVolumeConfiguration case can be reverted back to EFSVolumeConfiguration. It looks like the this pull request reverted the casing.

Workarounds

1. Update the synthesized template with proper casing and use it for creating the stacks.

2. In CDK code, override the volumes property as below.

const volumes = [
      {
        Name: "myefs",
        EFSVolumeConfiguration: {
          FilesystemId: filesystem.fileSystemId,
          TransitEncryption: "ENABLED",
        },
      },
    ]
    const cfnbluetaskDefinition = bluetaskDefinition.node.defaultChild as ecs.CfnTaskDefinition;

    // Overriding volumes object so that EFSVolumeConfiguratoin works for stack update.
    cfnbluetaskDefinition.addOverride('Properties.Volumes', volumes);

---

This is :bug: Bug Report

[mrpackethead]

I've discovered another quirk. when using GreenBlue Deploy. Its does appear that it a CF issue, not a CDK one.

https://github.com/aws-samples/aws-reinvent-2019-trivia-game/issues/299

[github-actions[bot]]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.