When deploying ECS service with blue/green deployment, task definition and target group changes can't be ignored

[morinrod-cardioscan]

I created a stack deploying ECS service with blue/green deployment(CodeDeploy) using aws.ecs.Service(). After the creation of the service in AWS, when running "pulumi up" again without any change to the existing service, Pulumi detects there was an update to the service.(because the TG or task definition revision was changed on AWS) Pulumi up -> the service is created. Running Pulum up without making any changes -> an update is detected on the service.

Because of AWS limitations, the update fails : “InvalidParameterException: Unable to update network parameters on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment.” I tried using the "ignore changes" and transformations properties when creating the service but it didn't help.

The issue forces me to delete all existing services and recreate them, If I need to make additional changes to the stack.

Related issues: pulumi/pulumi-awsx#856 https://github.com/pulumi/pulumi-aws/issues/1982

The service code:

const UsersService = new aws.ecs.Service("UsersService", {
    name: 'UsersService',
    cluster: ecs_cluster.id,
    schedulingStrategy: 'REPLICA',
    taskDefinition: 'Users-TD:1',
    platformVersion: 'LATEST',
    deploymentController:{
        type:'CODE_DEPLOY'
     } ,
    deploymentMaximumPercent: 200,
    deploymentMinimumHealthyPercent: 100,
    desiredCount: 2,
    launchType: 'FARGATE',

    networkConfiguration:{
        subnets: [subnet1,subnet2],
        assignPublicIp: false,
        securityGroups: [sg]
    },
    healthCheckGracePeriodSeconds: 20,
    loadBalancers:[
        {
            containerName:'UsersContainer',
            containerPort:'80',
            targetGroupArn: UsersTG1.arn

        }
    ],   
},
{
    ignoreChanges: ['desiredCount',
                'loadBalancers',
                'taskDefinition',
                ],
    transformations: [
        args => {
        if (args.type === 'aws:ecs/service:Service') {
            return {
            props: args.props,
            opts: pulumi.mergeOptions(args.opts, {
                ignoreChanges: [
                'desiredCount',
                'loadBalancers',
                'taskDefinition',
                ],
            }),
            };
        }
        return undefined;
        },
    ]
});

[leezen]

Does pulumi refresh help with this? That would sync the actual state to your current state and then I would expect the subsequent update to be correct once the code is also updated to reflect the state?

[morinrod-cardioscan]

I tried running pulumi refresh, when running pulumi up again without making any changes, there are still changes detected on the service. Changes were also detected on the listener and deployment group - since there was a deployment(blue\green), the target group has been changed.

[diegosasw]

is there any update on this?

What's the recommended approach when we have, for example, an AWS Fargate with services and task definitions, and we want to deploy new version of a service, but ignore existing ones?

I thought of running pulumi up with some env variables, and then, if the env variable was not provided, ignore containerDefinitions, for example. But that's not ideal, as other resources are created. And, of course, I don't want to delete the applications that don't require new deployment.

[t0yv0]

Possibly related: https://github.com/pulumi/pulumi-terraform-bridge/issues/1756

[dan-cooke]

I've spent most of the day on this, and I have hacked a solution together. From looking at old terraform issues / CDK issues - its quite disappointing that we have to hack around AWS on this.

In an ideal world AWS ECS should just turn the task definition update into a new deployment on code deploy, I found a relevant issue on CloudFormation https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1529

But I don't believe pulumi or terraform utilise CloudFormation - so I'm resorting to a hacky method while I investigate further.

But here's what I'm doing

1. Create ECS service with a hardcoded image, nginx:latest is what I'm using

   
   const service = new awsx.ecs.FargateService(
   'templi-api',
   {
   cluster: baseInfra.requireOutput('clusterArn'),
   enableExecuteCommand: true,
   
   deploymentController: {
     type: 'CODE_DEPLOY',
   },
   networkConfiguration: {
     securityGroups: [apiSecurityGroup.id],
     assignPublicIp: true,
     subnets: baseInfra.requireOutput('vpcPublicSubnetIds'),
   },
   continueBeforeSteadyState: true,
   
   // TODO: this is hardcoded as nginx due to a bug with pulumi not being able
   // to ignore changes on the task definition
   // As our deployments are handled by code deploy, we should ignore changes on the task definition
   taskDefinition: 'nginx:latest',
   loadBalancers: [
     {
       containerPort: 80,
       containerName: 'templi-api',
       targetGroupArn: blueTargetGroup.arn,
     },
   ],
   },
   {
   ignoreChanges: ['taskDefinition'],
   }
   );

2. Use your latest task definition to trigger a `pulumi/command` to go through the `aws deploy` CLI
```.ts
const deployment = latestTaskDefinitionArn.apply((arn) => {
  const appSpecJson = JSON.stringify({
    version: 0.0,
    Resources: [
      {
        TargetService: {
          Type: 'AWS::ECS::Service',
          Properties: {
            TaskDefinition: arn,
            LoadBalancerInfo: {
              ContainerName: 'templi-api',
              ContainerPort: 80,
            },
          },
        },
      },
    ],
  });

  const cliInputJson = JSON.stringify({
    applicationName: 'templi-api',
    deploymentGroupName: 'api-deployment-group',
    revision: {
      revisionType: 'AppSpecContent',
      appSpecContent: {
        content: appSpecJson,
      },
    },
  });

  return new local.Command('create-deployment', {
    create: `aws deploy create-deployment --cli-input-json '${cliInputJson}'`,
    environment: {
      CACHE_BUSTER: new Date().getTime().toString(),
    },
  });
});

export const deploymentId = deployment.stdout;