Document updating Fargate Service Task Definition using CODE_DEPLOY deployment controller

[dan-cooke]

What happened?

It is impossible to update an existing Fargate service using pulumi/aws or pulumi/awsx with a deploymentController configured as CODE_DEPLOY

If you attempt this, you will receive the following error message

Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment

I am unsure where this issue is coming from, I notice previously closed issues on the subject to do with upstream TF provider

https://github.com/pulumi/pulumi-aws/issues/1961

Related issues: https://github.com/pulumi/pulumi-awsx/issues/856 https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1529 https://github.com/aws/aws-cdk/issues/23564

Example

Given the following Fargate Service task definition

const service = new awsx.ecs.FargateService(
  'templi-api',
  {
    cluster: baseInfra.requireOutput('clusterArn'),
    deploymentController: {
      type: 'CODE_DEPLOY',
    },
    assignPublicIp: true,
    continueBeforeSteadyState: true,

    taskDefinition: latestTaskDefinition.arn,
    loadBalancers: [
      {
        containerPort: 80,
        containerName: 'templi-api',
        targetGroupArn: blueTargetGroup.arn,
      },
    ],
  },
  {
    ignoreChanges: ['taskDefinition'],
  }
);

With a task definition that changes , (ie. docker image update)

You will be unable to run pulumi up more than once.

Output of pulumi about

CLI
Version 3.107.0 Go Version go1.22.0 Go Compiler gc

Plugins NAME VERSION nodejs unknown

Host
OS arch Version
Arch x86_64

This project is written in nodejs: executable='/run/user/1000/fnm_multishells/23785_1713179166746/bin/node' version='v20.9.0'

Current Stack: templi/templi-api/prod

TYPE URN pulumi:pulumi:Stack urn:pulumi:prod::templi-api::pulumi:pulumi:Stack::templi-api-prod pulumi:providers:pulumi urn:pulumi:prod::templi-api::pulumi:providers:pulumi::default pulumi:providers:aws urn:pulumi:prod::templi-api::pulumi:providers:aws::default_6_27_0 aws:iam/role:Role urn:pulumi:prod::templi-api::aws:iam/role:Role::api-code-deploy-role aws:acm/certificate:Certificate urn:pulumi:prod::templi-api::aws:acm/certificate:Certificate::api-cert aws:codedeploy/application:Application urn:pulumi:prod::templi-api::aws:codedeploy/application:Application::templi-api aws:rds/parameterGroup:ParameterGroup urn:pulumi:prod::templi-api::aws:rds/parameterGroup:ParameterGroup::templi-database-parameter-group aws:route53/record:Record urn:pulumi:prod::templi-api::aws:route53/record:Record::api-cert-validation-record aws:acm/certificateValidation:CertificateValidation urn:pulumi:prod::templi-api::aws:acm/certificateValidation:CertificateValidation::api-cert-validation pulumi:pulumi:StackReference urn:pulumi:prod::templi-api::pulumi:pulumi:StackReference::templi/templi-infra/prod pulumi:providers:awsx urn:pulumi:prod::templi-api::pulumi:providers:awsx::default_2_5_0 aws:ec2/securityGroup:SecurityGroup urn:pulumi:prod::templi-api::aws:ec2/securityGroup:SecurityGroup::templi-api-sg aws:lb/targetGroup:TargetGroup urn:pulumi:prod::templi-api::aws:lb/targetGroup:TargetGroup::blue-target-group aws:rds/subnetGroup:SubnetGroup urn:pulumi:prod::templi-api::aws:rds/subnetGroup:SubnetGroup::templi-database-subnet-group aws:iam/role:Role urn:pulumi:prod::templi-api::aws:iam/role:Role::templi-api-task-role aws:lb/targetGroup:TargetGroup urn:pulumi:prod::templi-api::aws:lb/targetGroup:TargetGroup::green-target-group aws:ec2/securityGroup:SecurityGroup urn:pulumi:prod::templi-api::aws:ec2/securityGroup:SecurityGroup::templi-database-sg aws:rds/instance:Instance urn:pulumi:prod::templi-api::aws:rds/instance:Instance::templi-database awsx:ecr:Image urn:pulumi:prod::templi-api::awsx:ecr:Image::templi-api pulumi:providers:aws urn:pulumi:prod::templi-api::pulumi:providers:aws::default_6_9_0 pulumi:providers:docker urn:pulumi:prod::templi-api::pulumi:providers:docker::default_4_5_0 docker:index/image:Image urn:pulumi:prod::templi-api::awsx:ecr:Image$docker:index/image:Image::6f49d865-container awsx:lb:ApplicationLoadBalancer urn:pulumi:prod::templi-api::awsx:lb:ApplicationLoadBalancer::templi-lb aws:lb/loadBalancer:LoadBalancer urn:pulumi:prod::templi-api::awsx:lb:ApplicationLoadBalancer$aws:lb/loadBalancer:LoadBalancer::templi-lb aws:lb/targetGroup:TargetGroup urn:pulumi:prod::templi-api::awsx:lb:ApplicationLoadBalancer$aws:lb/targetGroup:TargetGroup::templi-lb awsx:ecs:FargateService urn:pulumi:prod::templi-api::awsx:ecs:FargateService::templi-api aws:ecs/service:Service urn:pulumi:prod::templi-api::awsx:ecs:FargateService$aws:ecs/service:Service::templi-api aws:lb/listener:Listener urn:pulumi:prod::templi-api::aws:lb/listener:Listener::blue-listener aws:route53/record:Record urn:pulumi:prod::templi-api::aws:route53/record:Record::api.templi.ai aws:lb/listener:Listener urn:pulumi:prod::templi-api::aws:lb/listener:Listener::green-listener awsx:ecs:FargateTaskDefinition urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition::templi-api aws:iam/role:Role urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:iam/role:Role::templi-api-execution aws:cloudwatch/logGroup:LogGroup urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:cloudwatch/logGroup:LogGroup::templi-api aws:codedeploy/deploymentGroup:DeploymentGroup urn:pulumi:prod::templi-api::aws:codedeploy/deploymentGroup:DeploymentGroup::api-code-deploy aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:iam/rolePolicyAttachment:RolePolicyAttachment::templi-api-execution-9a42f520 aws:ecs/taskDefinition:TaskDefinition urn:pulumi:prod::templi-api::awsx:ecs:FargateTaskDefinition$aws:ecs/taskDefinition:TaskDefinition::templi-api pulumi:providers:command urn:pulumi:prod::templi-api::pulumi:providers:command::default_0_9_2 command:local:Command urn:pulumi:prod::templi-api::command:local:Command::create-deployment

Found no pending operations associated with prod

Backend
Name pulumi.com URL https://app.pulumi.com/dan-cooke User dan-cooke Organizations dan-cooke, templi Token type personal

Pulumi locates its logs in /tmp by default

Additional context

If this falls into the wontfix, working as expected category - maybe we can document this somewhere? I've spent quite some time today trying to get blue/green deployments working with ECS + CodeDeploy via Pulumi, and the experience has been extremely painful to say the least.

The suggested workaround in the meantime seems to be to use ignoreChanges on your ECS service task definition, but unfortunately that is also not working atm. https://github.com/pulumi/pulumi-awsx/issues/856

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[corymhall]

@dan-cooke I saw your comment on https://github.com/pulumi/pulumi-awsx/issues/856#issuecomment-2057512670. It sounds like there might be two issues:

1. awsx does not allow you to ignoreChanges (tracked through https://github.com/pulumi/pulumi-awsx/issues/856)
2. Inadequate documentation on how to perform ECS blue/green deployments (we can track with this issue)

There also might be an opportunity for a new component for triggering code deploy deployments similar to the cdk-ecs-codedeploy CDK construct. Although an argument might also be made that this is a good scenario to use the automation api.

After the first deployment, all subsequent updates have to go through the Code Deploy service, but I agree that we could make that piece much easier.

[dan-cooke]

hey @corymhall thanks for your reply!

I agree there are a few interconnected issues here, and I like your suggestion to use this issue for documentation updates - I think a piece on blue/green deployments would go a long way.

It would be useful to see how we should handle this in Pulumi, for example if the solution is to ignore taskDefinitions, then how do we subsequently update other information about our taskDefinition? ie. memory cpu etc. surely this problem has been encountered in Terraform and CloudFormation too.

Its a tricky one, because it appears to be a fundamental issue with AWS ECS x CodeDeploy rather than an IaC specific problem.

[corymhall]

Its a tricky one, because it appears to be a fundamental issue with AWS ECS x CodeDeploy rather than an IaC specific problem.

100% agree with this. I'm curious what the killer CodeDeploy features are in your opinion? Part of me wants to find a solution to this that doesn't involve CodeDeploy and just uses ECS native features.

[dan-cooke]

Good question, and honestly its something I hadn't even thought about. Perhaps the golden path here is ECS native.

But off the top of my head, the features that I love from CodeDeploy

• Rollback on CloudWatch alert - I use this to rollback on e2e test failures
• Traffic re-routing built in, easy to start with 0% and only begin routing traffic when tests pass
• UI to manually route traffic / rollback - I use this quite a bit currently as a kind of final sanity check before I release

Some of that would be quite difficult (impossible for the UI) to achieve with ECS on its own.

But I am definetly willing to explore the option, the UI is optional - eventually I will be moving to fully automated releases anyway.