Using the aws_serverlessapplicationrepository_cloudformation_stack resource works the first time, but subsequent updates fail.

cdhesse commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.5

Affected Resource(s)

aws_serverlessapplicationrepository_cloudformation_stack

Terraform Configuration Files

data aws_serverlessapplicationrepository_application directory_app {
  application_id   = var.app_id
  semantic_version = var.app_semver
}

resource aws_serverlessapplicationrepository_cloudformation_stack directory_index_stack {
  count = var.disabled || ! var.enable_default_directory_index ? 0 : 1
  name  = "${var.stack_name}-${var.stage}-directory-index"

  tags = var.tags

  application_id = data.aws_serverlessapplicationrepository_application.directory_app.application_id
  capabilities   = data.aws_serverlessapplicationrepository_application.directory_app.required_capabilities

  parameters = {
    Version = var.version
  }
}

Expected Behavior

Running 2 times should not cause an error.

Actual Behavior

Doing a terraform apply the second time results in an error: Error: error creating Serverless Application Repository CloudFormation Stack (arn:aws:cloudformation:::stack//) change set: unexpected state 'FAILED', wanted target 'CREATE_COMPLETE'. last error: %!s()

Looking in cloudformation, the change set is Status: Failed, Status Reason: No updates are to be performed.

Steps to Reproduce

terraform apply (my example is using Lambda at edge functions if this makes a difference)
terraform apply a second time

Important Factoids

References

anGie44 commented 3 years ago

Hi @cdhesse , thank you for raising this issue. To further investigate the behavior you're seeing, do you mind describing what changes in your configuration between the first and second apply? also, if you have debug logs readily available or can reproduce the behavior in the meantime with the env TF_LOG=debug, it would be greatly appreciated to add them to the description when you have a chance :)

cdhesse commented 3 years ago

Thanks @anGie44 - the point is nothing is changed from the first run to the second... aka - nothing changed... simply run terraform apply 2x's in a row and i get the error. The fact that it is saying that the changeset failed due to no changes found, is accurate. Seems to me that that the "-no-fail-on-empty-changeset" parameter, which is part of the API is set to "-fail-on-empty-changeset".

cdhesse commented 3 years ago

Following up here to provide sample code to re-create this error with the AWS provided standard "hello-world" application.

terraform {
  required_version = "~> 0.13"
  required_providers {
    aws = "~> 3.18"
  }
}

provider "aws" {
  region = "us-east-1"
}

data aws_serverlessapplicationrepository_application app {
  application_id   = "arn:aws:serverlessrepo:us-east-1:077246666028:applications/hello-world"
  semantic_version = "1.0.4"
}

resource aws_serverlessapplicationrepository_cloudformation_stack hello_world_stack {
  name = "serverless-hello-world-test"

  application_id = data.aws_serverlessapplicationrepository_application.app.application_id
  capabilities   = data.aws_serverlessapplicationrepository_application.app.required_capabilities
  parameters = {
    IdentityNameParameter: "MyName"
  }
}

Run terraform apply -> this succeeds
After success, change nothing, simply run terraform apply a second time. This FAILS.

Error aws_serverlessapplicationrepository_cloudformation_stack.hello_world_stack: Modifying... [id=arn:aws:cloudformation:us-east-1:XXXXXXXXX:stack/serverlessrepo-serverless-hello-world-test/7a6ef230-35a6-11zb-98ff-0ab512dce13f]

Error: error creating Serverless Application Repository CloudFormation Stack (arn:aws:cloudformation:us-east-1:XXXXXXXXX:stack/serverlessrepo-serverless-hello-world-test/7a6ef230-35a6-11zb-98ff-0ab512dce13f) change set: unexpected state 'FAILED', wanted target 'CREATE_COMPLETE'. last error: %!s()

Looking in Cloudformation you can see the error in the ChangeSet:

Status FAILED Description

Status reason No updates are to be performed. Created time 2020-12-07 11:09:31 UTC-0500 Execution status UNAVAILABLE

anGie44 commented 3 years ago

Hi @cdhesse, apologies for the delay! Seems this may be a two-part problem but the most straightforward way to work around this in the meantime would be to set capabilities = ["CAPABILITY_IAM"] in your config, and that should eliminate the need for the second terraform apply.

For more background, it looks like data.aws_serverlessapplicationrepository_application.app doesn't return any values in required_capabilities and instead the aws_serverlessapplicationrepository_cloudformation_stack.hello_world_stack resource returns the value CAPABILITY_IAM as seen in this terraform plan (after the first apply), resulting in a perpetual diff:

  # aws_serverlessapplicationrepository_cloudformation_stack.hello_world_stack will be updated in-place
  ~ resource "aws_serverlessapplicationrepository_cloudformation_stack" "hello_world_stack" {
      ~ capabilities     = [
          - "CAPABILITY_IAM",
        ]
        id               = "arn:aws:cloudformation:us-east-1:xxxxxxxxxx:stack/serverlessrepo-serverless-hello-world-test/bbaaf210-38e9-11eb-94f7-0acfa743837d"
        name             = "serverless-hello-world-test"
        tags             = {}
        # (4 unchanged attributes hidden)
    }

This implies the AWS Cloudformation API computes the capabilities value, even when one is not provided, so that's a bug in the provider such that the capabilities field should be considered an Optional/Computed not Required field.

The second issue regarding the error seen after terraform attempts to Modify the resource: it could be a valid error from the AWS Cloudformation API as this troubleshooting error page describes (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-no-updates-to-perform). We could follow-up with AWS support to see how they recommend making just a change to the capabilities field or mark the field as ForceNew as well if we can't update it after creation.

Hope this helps in the meantime @cdhesse!

cdhesse commented 3 years ago

Thanks for the workaround. Indeed I confirm that hard coding the capabilities to ["CAPABILITY_IAM"] allows it to work on subsequent apply runs. I thought the point was to grab the required capabilities from the serverless application repo definition in case they get updated from version to version. Is that not a right understanding?

capabilities = data.aws_serverlessapplicationrepository_application.app.required_capabilities

anGie44 commented 3 years ago

hi @cdhesse, indeed in general you could grab the required_capabilities attribute from the data-source but it seems in the example that AWS provides, that serverless application doesn't have any set :/ . that field is set to an empty array [] in terraform and here's why I'm seeing in the Console (after being redirected from Serverless App Repository page):

cdhesse commented 3 years ago

So - interesting that there seems to be a second issue. For applications that require parameters (my real use case, not the hello-world one)... once i hard coded the CAPABILITY_IAM for this stack, I noticed a new bug. It seems that the parameters are not being persisted. Terraform refresh pulls back 4 parameters, even though there are 7 parameters in the configuration. I've not noticed anything strange about the parameters being dropped vs. kept, but something to look into. I'll try to dig deeper but the result is that this component is still not working for me given this second issue.

moko-001 commented 3 years ago

Hi, just checking to see if there have been any updates on this, please? we are currently facing the exact same issue with using aws_serverlessapplicationrepository_cloudformation_stack, we also use parameters and have not been able to get a second deployment to work.

simonhanmer commented 3 years ago

Also seeing this issue.

We've deployed a stack using

resource "aws_serverlessapplicationrepository_cloudformation_stack" "salesforceIntegrationLambda" {
    name            = "salesforceIntegrationLambda"
    application_id  = "arn:aws:serverlessrepo:us-west-2:821825267871:applications/AmazonConnectSalesforceLambda"

    capabilities = [ 
        "CAPABILITY_IAM",
        "CAPABILITY_NAMED_IAM"
    ]

    parameters = {
      ...
   }
}

which creates a cloudformation stack called serverlessrepo-salesforceIntegrationLambda.

However, on the second terraform apply, we get a failure. Checking CloudTrail, I can see that a cloudformation.DescribeStacks API call is failing with the error Stack with id aws-serverless-repository-salesforceIntegrationLambda does not exist - it seems to be looking for the wrong stack name?

DanyC97 commented 2 years ago

following up based on same example provided by @cdhesse , i've run the below and i didn't get the error on 2nd time when run terraform apply

terraform {
  required_version = "~> 0.13"
  required_providers {
    aws = "~> 3.50.0"
  }
}

provider "aws" {
  region = "us-east-1"
  profile = "xxx"
}

data aws_serverlessapplicationrepository_application app {
  application_id   = "arn:aws:serverlessrepo:us-east-1:077246666028:applications/hello-world"
  semantic_version = "1.0.4"
}

resource aws_serverlessapplicationrepository_cloudformation_stack hello_world_stack {
  name = "serverless-hello-world-test"

  application_id = data.aws_serverlessapplicationrepository_application.app.application_id
  capabilities = [
        "CAPABILITY_IAM",
        "CAPABILITY_NAMED_IAM"
  ]
  parameters = {
    IdentityNameParameter: "MyName"
  }
}

not sure which version @simonhanmer been running with but you can see my aws provider version above. Hope that helps a bit

kgns commented 1 year ago

So - interesting that there seems to be a second issue. For applications that require parameters (my real use case, not the hello-world one)... once i hard coded the CAPABILITY_IAM for this stack, I noticed a new bug. It seems that the parameters are not being persisted. Terraform refresh pulls back 4 parameters, even though there are 7 parameters in the configuration. I've not noticed anything strange about the parameters being dropped vs. kept, but something to look into. I'll try to dig deeper but the result is that this component is still not working for me given this second issue.

I can confirm that this problem still exists in the latest version of the provider even after almost 2 year. As @cdhesse said, if you have more than 4 parameters, somehow the plan only refreshes up to 4 parameters from current stack that exists in AWS, and the rest of the parameters are treated as new, hence a diff is calculated. But when you apply the plan, since the diff that was calculated is not the reality, apply errors out with change set: unexpected state 'FAILED', wanted target 'CREATE_COMPLETE'. last error: No updates are to be performed error

bobk81 commented 1 year ago

anGie44 and kgns I can also confirm this is still a problem. I am deploying "cloudfront-authorization-at-edge" and get the same errors, I also started off with using this option.

capabilities = data.aws_serverlessapplicationrepository_application.app.required_capabilitie

I then changed it to

capabilities = [ "CAPABILITY_IAM", "CAPABILITY_RESOURCE_POLICY" ] I also checked, and unlike the above suggestion, both capabilities are listed.

Please advise if anyone has been able to find a working, workaround for this.

troyready commented 1 year ago

I'm pretty sure NoEcho CFN parameters are triggering this same error as well:

          ~ "ParameterName"        = "****" -> "ValueFromTerraform"

Terraform thinks the resource needs to be updated but on apply the CFN change-set reports an error of no changes to be performed.

yesteph commented 1 year ago

I confirm Terraform try to update the cloudformation stack but as there is no change, the AWS Cloud formation Change Set is empty then enters in Failed status.

When I look at the Terraform state file, It seems that some credentials parameters are stored with "****"

taiseii commented 1 year ago

My work around to this issue. just use the local .json or .yml CloudFormation template with terraform aws_cloudformationn_stack resource. Then terraform apply works fine without destroying the CloudFormation stack resource.

bobk81 commented 1 year ago

My work around to this issue. just use the local .json or .yml CloudFormation template with terraform aws_cloudformationn_stack resource. Then terraform apply works fine without destroying the CloudFormation stack resource.

I have done the same for the time being. The only problem with this is now I need to have that cloudformation template uploaded to an S3 bucket first, as it is too large. If would be great if it could all work as it is supposed to, but the workaround works for now.

Amir-Ahmad commented 1 year ago

My temporary solution for this was to add a tag with a random string. this forces an update on every apply though, so it's not ideal.

resource "aws_serverlessapplicationrepository_cloudformation_stack" "some_app" {
  tags = {
    RollMe = random_string.rollme.result
  }
}

resource "random_string" "rollme" {
  length  = 16
  special = false

  keepers = {
    uuid = "${uuid()}"
  }
}

vgw-chriskruger commented 1 year ago

Very annoying issue.

arckdash commented 11 months ago

Experiencing the same issue as everyone else on this thread, has anyone managed to find a workaround for this?

In my case we're using terraform cloud for state management and triggers from GHA.

arckdash commented 11 months ago

My temporary solution for this was to add a tag with a random string. this forces an update on every apply though, so it's not ideal.
resource "aws_serverlessapplicationrepository_cloudformation_stack" "some_app" {
  tags = {
    RollMe = random_string.rollme.result
  }
}

resource "random_string" "rollme" {
  length  = 16
  special = false

  keepers = {
    uuid = "${uuid()}"
  }
}

When testing this workaround i get the error:

Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for
│ aws_serverlessapplicationrepository_cloudformation_stack.datadog_rds_enhanced
│ to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .tags_all: new element "RollMe" has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

-_- any suggestions?

tbalzer commented 2 weeks ago

After some tedious debugging we found two things that don't work as expected here:

The terraform state only reflects non-default values of the app parameters, so if any parameters are set to the default value this will conflict as the resource will constantly try to remediate this perceived difference
The terraform state has the value of "secret"/NoEcho parameters represented as *s which also leads to constant differences (see https://github.com/hashicorp/terraform-provider-aws/issues/16485#issuecomment-1324478395)

Our solution/workaround/hack:

Do not set parameters to their default value, just omit them
Add secret/NoEcho parameters to the lifecycle ignore_changes block (beware if you actually want to change those!)

UPDATE: ignore_changes also only works if there won't ever be changes to the resource. Any changes will otherwise lead to errors as secret/NoEcho parameter values will be set to *s then. Only option then is to introduce some randomly generated tag which is ugly but kind-of works.

nickjmv commented 5 days ago

@tbalzer, could you provide an example on how to set these in the ignore changes block? I tried the following but this results in an error.

  lifecycle {
    ignore_changes = [
      parameters[0].GoogleAdminEmail
    ]
  }

Error:

Can't access attributes on a primitive-typed value (string).

tbalzer commented 4 days ago

@nickjmv This works with string indexes, e.g. parameters["GoogleAdminEmail"] works fine

hashicorp / terraform-provider-aws