‼️ NOTICE: last modified date does not match with the last modified date of the retrieved parameters

MrArnoldPalmer commented 3 years ago

Please add your +1 👍 to let us know you have encountered this

Status: INVESTIGATING

Overview:

Cloudformation recently deployed a change to allow dynamic references to ssm parameters within templates without needing to provide a version. This allows always referencing the latest value of a parameter within your template. This change had a bug that caused change set creation to fail when the parameter value was updated.

This breakage was caused by the references resolving to a version of the parameter that include a timestamp. Upon changeset creation, if the parameter has been updated, cloudformation sees the parameter name and the previously resolved timestamped version as different values and fails to update. They have since changed logic so that the strings resolved do not include timestamps, however if a template was successfully deployed during the time that the timestamp was being appended, a stack cannot be updated via changeset while referencing a dynamic parameter.

UPDATE:

It is clear that removing all dynamic ssm references doesn't cause users to get unstuck. Users have to manually deploy their stacks without a changeset to get past this. That being the case, for users with lots of stacks updating each manually is a pain. We are considering adding an option to the CLI to skip a changeset if this error occurs during deploy.

Complete Error Message:

Parameters: [PARAMETER_NAME] last modified date does not match with the last modified date of the retrieved parameters.

Workaround:

Attempt the following steps in order as a workaround:

Upgrade both your CLI and all your @aws-cdk/xyz modules to v1.103.0 - This removes dynamic SSM references and goes back to referencing them via CFN parameters.
Delete your stack and redeploy - If you are unable to do so because this is a production stack, and upgrading to v1.103 didn't fix the issue, please comment here with the following information and try number 3.
- The name of the parameter causing the failure
- An example of how you are referencing that parameter in code, IE through an enum value, manually specifying the param name etc.
Run cdk synth and then manually deploy this stack using the the aws cli or cfn console - This is not an ideal workaround for users who may have a lot of different stacks that need updating but in the meantime, this can unblock you if you need a stack deployed. We are working on figuring out another option that will allow skipping changeset creation via the aws cli.

Solution:

This issue only shows up for users who deployed stacks during a short period of time before cloudformation fixed some of these bugs. Because of that, it is hard for us to test ourselves if removal of dynamic param references (https://github.com/aws/aws-cdk/pull/14527) solves this. If it doesn't, another solution mentioned here is to update your stack without creating a changeset. We may be able to provide an option in the CLI to skip a changeset if this error is encountered. Since we don't have broken stacks to test this with, please let us know if this is or an additional fix is needed.

hermes-z commented 3 years ago

I have had this problem twice and after upgrading to v1.103 it seems to be solved. Thank you very much.

Urokhtor commented 3 years ago

We have this issue after upgrading to v1.103, but haven't encountered it before. We like to keep our dependencies up to date, so we have updated to every minor release in recent history.

The change set that failed has this parameter:

SsmParameterValueawsserviceecsoptimizedamiamazonlinux2recommendedimageidC96584B6F00A464EAD1953AFF4B05118Parameter:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id

That parameter is referenced as the ImageId of autoscaling group launch configuration. In the template currently deployed with CDK v1.102, ImageId is defined as "{{resolve:ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id}}".

Urokhtor commented 3 years ago

We had this happen on another stack that has not been updated to v1.103 yet. It's still using v1.102. I don't know if this helps, but in this stack the ImageId still uses "{{resolve:ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id}}" instead of passing it as a parameter.

nirradi commented 3 years ago

I have this issue, and I'm on 1.103

Failed to create ChangeSet cdk-deploy-change-set on production-ec2: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters.

I believe it has to do with the AMI

linux_ami = ec2.AmazonLinuxImage(generation=ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
                                 edition=ec2.AmazonLinuxEdition.STANDARD,
                                 virtualization=ec2.AmazonLinuxVirt.HVM,
                                 storage=ec2.AmazonLinuxStorage.GENERAL_PURPOSE
                                 )  # Indicate your AMI, no need a specific id in the region

eladb commented 3 years ago

@nirradi two quick things to validate:

Are you using both CLI version 1.103 and @aws-cdk/xxx 1.103?
Can you please share your CFN template that was synthesized

nirradi commented 3 years ago

@nirradi two quick things to validate:

Are you using both CLI version 1.103 and @aws-cdk/xxx 1.103?

Can you please share your CFN template that was synthesized

I see now that the two systems which I work on differ by the version. Its working on 1.101 (both python packages and cli) and its not working on 1.103 (both python packages and cli)

I'm not inclined to share the entire cfn, here are the diffs between 1.101 and 1.103

111c111,113
<         "ImageId": "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}",
---
>         "ImageId": {
>           "Ref": "SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter"
>         },
567a572,577
>   },
>   "Parameters": {
>     "SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter": {
>       "Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
>       "Default": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
>     }
569c579

MrArnoldPalmer commented 3 years ago

@nirradi if this is failing, cdk deploy should fail for both v1.102 and v1.103. The change in the template itself in v1.103 is supposed to allow cdk update to work by removing the {{resolve:ssm/...}} syntax.

With v1.103.0, can you give us the output of cdk ls -v, look through it and remove any info you don't want to share.

blaenk commented 3 years ago

Mentioned this in the other issue but just a friendly reminder so it doesn't fall through the cracks: I'm happy to try this on the 2.0 line (I recently ported to from 1.x to 2 RC1) but I'm waiting for the 2.0 release to come through (last one was ~12 days ago).

rsyabuta commented 3 years ago

I resolved this issue on my EKS stacks by manually updating a stack as referenced here: https://github.com/aws/aws-cdk/issues/14467#issuecomment-833765356 with a stack template generated with 1.103.0. Subsequent cdk deploy commands worked as expected.

MrArnoldPalmer commented 3 years ago

@rsyabuta to clarify, cdk deploy on v1.103.0 was still failing with the param issue and required deploying without a change set?

rsyabuta commented 3 years ago

Yes that is correct. Here is the cdk diff output for reference:

Stack development
Parameters
[+] Parameter SsmParameterValue:--aws--service--eks--optimized-ami--1.19--amazon-linux-2--recommended--image_id:C96584B6-F00A-464E-AD19-53AFF4B05118.Parameter SsmParameterValueawsserviceeksoptimizedami119amazonlinux2recommendedimageidC96584B6F00A464EAD1953AFF4B05118Parameter: {"Type":"AWS::SSM::Parameter::Value<String>","Default":"/aws/service/eks/optimized-ami/1.19/amazon-linux-2/recommended/image_id"}

Resources
[~] AWS::AutoScaling::LaunchConfiguration Cluster/Compute/LaunchConfig ClusterComputeLaunchConfig23228ED8 replace
 └─ [~] ImageId (requires replacement)
     └─ @@ -1,1 +1,3 @@
        [-] "{{resolve:ssm:/aws/service/eks/optimized-ami/1.19/amazon-linux-2/recommended/image_id}}"
        [+] {
        [+]   "Ref": "SsmParameterValueawsserviceeksoptimizedami119amazonlinux2recommendedimageidC96584B6F00A464EAD1953AFF4B05118Parameter"
        [+] }

And the cdk deploy failure output:

development: creating CloudFormation changeset...

 ❌  development failed: Error: Failed to create ChangeSet cdk-deploy-change-set on development: FAILED, Parameters: [ssm:/aws/service/eks/optimized-ami/1.19/amazon-linux-2/recommended/image_id:7:1619988083400] last modified date does not match with the last modified date of the retrieved parameters.
    at /opt/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:227:11
    at processTicksAndRejections (node:internal/process/task_queues:94:5)
    at waitFor (/opt/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:188:20)
    at Object.waitForChangeSet (/opt/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:213:15)
    at Object.deployStack (/opt/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:260:32)
    at CdkToolkit.deploy (/opt/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/opt/local/lib/node_modules/aws-cdk/bin/cdk.ts:209:9)
Failed to create ChangeSet cdk-deploy-change-set on development: FAILED, Parameters: [ssm:/aws/service/eks/optimized-ami/1.19/amazon-linux-2/recommended/image_id:7:1619988083400] last modified date does not match with the last modified date of the retrieved parameters.

nirradi commented 3 years ago

cdk ls -v

@MrArnoldPalmer

CDK toolkit version: 1.103.0 (build bc13a66)
Command line arguments: {
  _: [ 'ls' ],
  v: 1,
  verbose: 1,
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  fail: false,
  long: false,
  l: false,
  '$0': 'cdk'
}
cdk.json: {
  "app": "python3 app.py"
}
cdk.context.json: {
  "availability-zones:account=********:region=eu-west-1": [
    "eu-west-1a",
    "eu-west-1b",
    "eu-west-1c"
  ]
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'python3 app.py',
  context: {},
  debug: false,
  assetMetadata: true,
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [],
  lookups: true
}
Determining if we're on an EC2 instance.
Looks like an EC2 instance.
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to ap-northeast-2
Resolving default credentials
Looking up default account ID from STS
Default account ID: ******
Setting "CDK_DEFAULT_ACCOUNT" environment variable to ******
context: {
  'availability-zones:account=*******:region=eu-west-1': [ 'eu-west-1a', 'eu-west-1b', 'eu-west-1c' ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': []
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'ap-northeast-2',
  CDK_DEFAULT_ACCOUNT: '******',
  CDK_CONTEXT_JSON: '{"availability-zones:account=*****:region=eu-west-1":["eu-west-1a","eu-west-1b","eu-west-1c"],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true,"aws:cdk:version-reporting":true,"aws:cdk:bundling-stacks":[]}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '11.0.0',
  CDK_CLI_VERSION: '1.103.0'
}

Malanius commented 3 years ago

As I've mentioned in https://github.com/aws/aws-cdk/issues/14467#issuecomment-833323448, I encounter this in the ec2.BastionHostLinux construct. Still present after an update to 1.103.0. Stack is in eu-west-1 region.

The reference in the stack is simple:

this.bastion = new ec2.BastionHostLinux(this, `Bastion`, {
      vpc: props.vpc,
      instanceName: `${props.name}-bastion`,
      securityGroup: bastionSg,
      subnetSelection: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
    });

Diff for the stack:

cdk diff vpc

Stack vpc (vpc)
Parameters
[+] Parameter SsmParameterValue:--aws--service--ami-amazon-linux-latest--amzn2-ami-hvm-x86_64-gp2:C96584B6-F00A-464E-AD19-53AFF4B05118.Parameter SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter: {"Type":"AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>","Default":"/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"}

Resources
[~] AWS::EC2::Instance Bastion/Bastion/Resource BastionBastion34649AD2 replace
 └─ [~] ImageId (requires replacement)
     └─ @@ -1,1 +1,3 @@
        [-] "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
        [+] {
        [+]   "Ref": "SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter"
        [+] }
[~] AWS::EC2::EIP Bastion/BastionEIP BastionBastionEIP54D072EC may be replaced
 └─ [~] InstanceId (may cause replacement)
     └─ [~] .Ref:
         ├─ [-] BastionBastion34649AD2
         └─ [+] BastionBastion34649AD2 (replaced)

Fails with:

vpc (vpc): deploying...
vpc: creating CloudFormation changeset...

 ❌  vpc (vpc) failed: Error: Failed to create ChangeSet cdk-deploy-change-set on -vpc: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters.
    at /home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:227:11
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at waitFor (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:188:20)
    at Object.waitForChangeSet (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:213:15)
    at Object.deployStack (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:260:32)
    at CdkToolkit.deploy (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/bin/cdk.ts:209:9)
Failed to create ChangeSet cdk-deploy-change-set on -vpc: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters.

The workaround with manually updating stack from generated template by 1.103.0 worked for me. Probably just temporarily deleting the offending bastion resource could work as well, but this way I at least avoided updating all the stacks where Bastion is mentioned.

cseickel commented 3 years ago

I had this issue repeatedly on 1.101 and 1.102 and my solution was to delete and redeploy the stack when it happened. The problem went away on it's own before this issue was opened without me having to upgrade. I believe the last time I hit it was May 6. I have deployed at least 20 updates in the last few days on that 1.102 stack with no issues.

blaenk commented 3 years ago

Does the new 2.0.0-rc.3 include the fix in 1.103? EDIT: Just saw the release notes that it does seem to include https://github.com/aws/aws-cdk/issues/14527

I have cdk and aws-cdk-lib updated to it, but I'm still running into the issue.

Error:

 ❌  Control failed: Error: Failed to create ChangeSet cdk-deploy-change-set on Control: FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:59:1619746225127] last modified date does not match with the last modified date of the retrieved parameters.

I think the parameter in question is this one:

[~] AWS::AutoScaling::LaunchConfiguration Cluster/Capacity/LaunchConfig ClusterCapacityLaunchConfigC44566A4 replace
 ├─ [~] ImageId (requires replacement)
 │   └─ @@ -1,1 +1,3 @@
 │      [-] "{{resolve:ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id}}"
 │      [+] {
 │      [+]   "Ref": "SsmParameterValueawsserviceecsoptimizedamiamazonlinux2recommendedimageidC96584B6F00A464EAD1953AFF4B05118Parameter"
 │      [+] }

However, I'm not explicitly referencing this myself AFAIK. Just double checked and I can't see anywhere where I am referencing anything like this.

I tried updating the stack via the web console, using the existing template without making any changes at all (is that the right process?) but I got the error:

Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:59:1619746225127] were updated before values were used

TLadd commented 3 years ago

Also ran into this with the ec2.BastionHostLinux construct. Since we only use this for connecting to resources within our vpc and it's not actively receiving usage all the time, I was able to comment it out, deploy, add it back, deploy and the issue went away.

MrArnoldPalmer commented 3 years ago

Updated the issue to include manually deploying your stack without a changeset as a workaround while we figure out if/how to allow users to do this via CDK. For users who have CI/CD deploying a high volume of stacks this obviously isn't ideal and we are figuring out if we want to add a way for the CDK CLI to deploy without a changeset and if that is good enough. If people are using the codepipeline stack deploy actions with a changeset this won't cover them and I'm not sure if there is another way to let them easily workaround this.

andreialecu commented 3 years ago

In my case I just upgraded to 1.103.0 earlier today, and noticed a diff like the one mentioned in: https://github.com/aws/aws-cdk/issues/14629#issuecomment-840278777

The deployment finished fine. However, the very next deployment failed.

  EcsCluster failed: Error: Failed to create ChangeSet cdk-deploy-change-set on EcsCluster: FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:59:1619744777312] last modified date does not match with the last modified date of the retrieved parameters.

So it seems that actually upgrading to 1.103.0 broke this.

~Note that this stack hasn't been deployed/updated since March so the mention in the OP that this was a bug in a certain time window seems suspect. See timestamps:~

The "faulty" parameter seems to be added automatically as part of creating an ECS cluster. It is not something I'm adding.

sundeqvist commented 3 years ago

We are running into this after upgrading to 1.103.0 today, seemingly caused by usage of EcsOptimizedImage.amazonLinux2. We've run several deploys over the last few days with no errors until we upgraded to 1.103.0.

Error is FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:59:1619744777312] last modified date does not match with the last modified date of the retrieved parameters.

Updating the stack with a 1.103.0 generated template fixed the issue for us.

MrArnoldPalmer commented 3 years ago

@andreialecu I don't see how that is possible. What version were you on when the stack was last deployed? Can you also verify that you were using both the CLI and @aws-cdk/* libraries with version 1.103.0? It appears that not only dynamic references were broken, but all SSM param references including those passed via CFN parameters. If that is the case, the timing of the breakage could be based purely on when these parameters were updated in relation to when stack updates occurred and when the feature was rolled out on CFN's side. This makes it really hard for us to figure out what users are affected and how exactly.

Are you able to attempt updating your stack manually as mentioned in the issue to workaround? I'm curious if that helps in your case.

andreialecu commented 3 years ago

@MrArnoldPalmer ~we were on 1.100.0 but the affected stack did not have any changes made to it since March. (As can be seen from the screenshot)~

It was only after updating to 103 that the diff appeared, as mentioned in my previous comment. Deploying it the first time worked. But any subsequent deploys have failed.

I haven't yet tried manually deploying.

andreialecu commented 3 years ago

Oops. My screenshot does show that it was actually previously deployed on the 5th of May, probably using 1.100.0. NOT in March. Sorry about that. Had a mental hiccup, I thought 05.05 was today 🤦🏻‍♂️

andreialecu commented 3 years ago

@MrArnoldPalmer I can confirm that manually updating the stack using the AWS console did work.

MrArnoldPalmer commented 3 years ago

Thank you @andreialecu for investigating!

Anyone else that is still blocked and having trouble with the workarounds please keep letting us know.

seawatts commented 3 years ago

Updated the issue to include manually deploying your stack without a changeset as a workaround while we figure out if/how to allow users to do this via CDK. For users who have CI/CD deploying a high volume of stacks this obviously isn't ideal and we are figuring out if we want to add a way for the CDK CLI to deploy without a changeset and if that is good enough. If people are using the codepipeline stack deploy actions with a changeset this won't cover them and I'm not sure if there is another way to let them easily workaround this.

Thanks for the response @MrArnoldPalmer 😃 I am using the CI/CD AWS CDK Pipeline to deploy my stacks. Just confirming what you're saying here, there is not currently a workaround? Would rolling back the cdk version work?

cseickel commented 3 years ago

I managed to go long time without this issue but it came back today. I upgraded from 1.101.0 to 1.104.0 but it did not fix it. This is for an ECS cluster and the parameter is implicit in the construct as opposed to anything I create in my own code: Failed to create ChangeSet cdk-deploy-change-set on alfresco-app-dev: FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:60:1620247830216] last modified date does not match with the last modified date of the retrieved parameters.

I used the workaround of cdk synth and manually uploaded the template via the console and that is working.

clemans commented 3 years ago

I used the workaround of cdk synth and manually uploaded the template via the console and that is working.

Upgrading from 1.102.0 to 1.104.0 did not solve my issue but the workaround provided by @cseickel worked.

MrArnoldPalmer commented 3 years ago

@seawatts unfortunately no, this isn't casued by anything in the CDK it's caused by an issue on the cloudformation side. The only workaround is to run cdk synth and then manually deploy the contents of cdk.out using the CLI or clouformation console. Doing so will allow you to skip changeset creation. Otherwise you would have to change your codepipeline actions to be CREATE_UPDATE instead of CHANGE_SET_REPLACE + CHANGE_SET_EXECUTE.

rahulpshah commented 3 years ago

Is the upgrade to cdk 1.103 necessary for the workaround? Would simply running cdk synth and applying the output of cdk.out on the UI work? Is it possible to roll back to known version? Current version: 1.101

kastningj commented 3 years ago

Unfortunately, none of the suggestions in this and other linked tickets have worked for me. I've tried:

1) upgrading cli and modules (currently on v1.104.0) Error: Failed to create ChangeSet cdk-deploy-change-set on my-stack-name: FAILED, Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:60:1620247830216] last modified date does not match with the last modified date of the retrieved parameters.

2) calling update-stack (without a change set): An error occurred (ValidationError) when calling the UpdateStack operation: Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:60:1620247830216] were updated before values were used

3) running cdk synth and then manually deploy my stack using the the aws cli Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Parameters: [ssm:/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id:60:1620247830216] last modified date does not match with the last modified date of the retrieved parameters.

Code: EcsOptimizedImage ecsOptimizedImage = EcsOptimizedImage.amazonLinux2(); AutoScalingGroup asg = new AutoScalingGroup(this, name + "-asg", AutoScalingGroupProps.builder() .instanceType(instanceType) .machineImage(ecsOptimizedImage) .blockDevices(blockDevices) .desiredCapacity(minCapacity) .maxCapacity(maxCapacity) .minCapacity(minCapacity) .keyName(clusterName) .groupMetrics(metrics) .vpc(vpc) .instanceMonitoring(isProduction ? Monitoring.DETAILED : Monitoring.BASIC) .allowAllOutbound(true) .role(createEC2InstanceRole(name + "-ec2-instance")) .build());

Are there any other suggestions to try other than deleting/recreating stacks?

clemans commented 3 years ago

Are there any other suggestions to try other than deleting/recreating stacks? @kastningj I have a similar CDK project using EcsOptimizedImage.amazonLinux2() and AutoScalingGroup. Uploading the cdk synth > cloudformation.yaml template in the AWS Management Console worked for me and I hope it does for you as well.

kastningj commented 3 years ago

@clemans Thanks! Your reply made me realize I hadn't tried uploading via the AWS Management Console, only command line. The upload via the console worked! Saves me from deleting / recreating stacks. Appreciate the help!

MrArnoldPalmer commented 3 years ago

@kastningj the workaround is deploy without creating a changeset, it looks like when you were using the CLI you were still going through changeset creation when what is required is doing update-stack directly. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-direct.html I see your second step now did try this, hasting reading on my part. I'm not sure what the difference between this and uploading via the console is though. In theory they should be the same.

Glad the console resolved this though that seems like the easiest if you only need to do a couple of stacks.

MrArnoldPalmer commented 3 years ago

Is the upgrade to cdk 1.103 necessary for the workaround? Would simply running cdk synth and applying the output of cdk.out on the UI work? Is it possible to roll back to known version? Current version: 1.101

@rahulpshah updating is not required to do the workaround, but it may prevent you from needing it as some users report v1.103.0 fixing the issue for them.

FlorinAsavoaie commented 3 years ago

Also getting this issue for obtainDefaultFluentBitECRImage from @aws-cdk/aws-ecs... This is really frustrating.

skotadia commented 3 years ago

I am using AWS SAM template. I believe, underneath, it is using aws cdk. I am getting same issue. I don't have an option of deletion of the stack or doing it manual. Do we have any workaround for aws sam CLI? Thanks

AbdoNile commented 3 years ago

I managed to find a fix for this with @Shogan . It is a "surgical" workaround , but will allow you to get the stack in an "updateable" state using CDK without recreating it.

get the new ssm parameter last modified date time stamp from aws console or using ssm cli get-parameter
convert the time into Unix format including milliseconds . I used https://time.lol for this. You should end up with something that looks like "1616080117000"
In AWS Console cloudformation screen , open the stack , click on "edit in designer"
the lower pan of the designed contains the template in YAML format
find and replace all instances of ""{{resolve:ssm:/ssm_parameter_name:1:OLD_TIME_STAMP}}" with ""{{resolve:ssm:/ssm_parameter_name:1:NEW UNIX_TIME_STAMP}}"
Save the template to a local YAML file to your computer
Close the Designer
From CloudFormation Console again , Click on Update Stack , Replace Existing stack , upload your template file that you downloaded from the designer.
Follow the wizard until the stack update is completed.

If all goes well , you should be able to deploy the stack using cdk going forward.

FlorinAsavoaie commented 3 years ago

I did something similar to @AbdoNile but simply replaced the parameters with the actual values. After that succeeded, I deployed again through CDK and it worked.

rix0rrr commented 2 years ago

This issue was a notification for a transient error. Due to the lack of recent engagement, I'll assume most people that were affected by this managed to unstick themselves by now. Closing.

github-actions[bot] commented 2 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-cdk