aws_cloudformation_stack sensitive parameters always trigger update

[ghost]

This issue was originally opened by @haidangwa as hashicorp/terraform#22939. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.12.9
+ provider.aws v2.28.1
+ provider.null v2.1.2

I have seen this since 0.11, however.

Terraform Configuration Files

resource "aws_cloudformation_stack" "chef_server" {
  name = local.namespace

  capabilities     = ["CAPABILITY_IAM"]
  disable_rollback = true

  parameters = {
    LDAPBindPassword             = data.aws_kms_secrets.chef.plaintext["bind_password"]
    DBPassword                         = data.aws_kms_secrets.chef.plaintext["db_password"]
  }

  template_url = var.cf_template_url

}

Expected Behavior

The first time this resource is taken through plan and apply, I expect an update. However, if I run plan and apply again right away, terraform will think that another update is necessary. This is not true.

Actual Behavior

terraform plan outputs that the resource will be updated in-place, but terraform apply does not update anything.

Snipped output:

  ~ resource "aws_cloudformation_stack" "chef_server" {
        capabilities     = [
            "CAPABILITY_IAM",
        ]
        disable_rollback = true
        id               = "arn:aws:cloudformation:us-west-2:698702875737:stack/chef-server-us-west-2-prod/eaf8a4b0-514f-11e8-9a70-50a686fc379a"
        name             = "chef-server-us-west-2-prod"
        outputs          = {
            "BootstrapAutoScaleGroup" = "chef-server-us-west-2-prod-BootstrapAutoScaleGroup-1MD2D8GE77IWO"
            "ChefLBDNSName"           = "internal-chef-server-us-west-2-prod-lb-1272281532.us-west-2.elb.amazonaws.com"
            "ChefTargetGroup"         = "arn:aws:elasticloadbalancing:us-west-2:698702875737:targetgroup/chef-ChefT-115ZYXNJF3EH2/1a4778ed8c286fb8"
            "FrontendAutoScaleGroup"  = "chef-server-us-west-2-prod-FrontendAutoScaleGroup-Q7UPOR74NI25"
        }
      ~ parameters       = {

          ~ "DBPassword"                         = "****" -> "REDACTED"

          ~ "LDAPBindPassword"                   = "****" -> "REDACTED"

        }

What I think is happening is that terraform the feature masking the actual passwords is creating a false positive when comparing the old password with the incoming "new" password from the run.

Steps to Reproduce

1. terraform plan
2. terraform apply
3. terraform plan
4. terraform apply

[acesir]

We are hitting this bug as well and it's fairly annoying as our cloudformation stacks are large and the triggered update forces the entire stack to print.

[anweiss]

+1 ... also hitting this

[hernan82arg]

+1

[mbevc1]

:+1:

[drogerschariot]

+1 Hitting this in parameters as well

[mbevc1]

Another thing I've noticed as looking into this, changing CFN template later on would always end up with plan changes, but if you destroy and re-apply that usually sorts it out if not using NoEcho parameters. :thinking:

[Apollorion]

Just wanted to add that this is still a thing on the latest AWS provider (3.46.0) and terraform 1.0.0.

[caleb15]

Also a problem with terraform v13 and v14.

This is also a security concern because the parameters may be sensitive (like the DBPassword example from OP) yet they get displayed in plaintext. In CI environments where many people have access to the plan this is a significant leak. Does anyone know of any workarounds that can be used so the sensitive value is not displayed?

I've emailed security at hashicorp.com to let them know about this issue.

[gdavison]

Hi everyone,

There are two related issues here, one with the provider's handling of NoEcho (sensitive) CloudFormation parameters, and one with sensitive values in Terraform itself.

CloudFormation allows parameters to be marked as NoEcho so that CloudFormation will treat them as sensitive. These values are returned in the AWS API as ****. The AWS provider is not handling these properly, and incorrectly shows a perpetual diff.

Secondly, Terraform is showing the values in console output. There are a few ways around this. When a Terraform resource has a known-sensitive value, we mark it as Sensitive in the provider. Unfortunately, in this case the parameters are defined in the CloudFormation template, so they are not known to us beforehand.

There are several workarounds that can be used in Terraform configurations, depending on where the sensitive values are coming from:

• Terraform input values can be marked sensitive, which will mask them
• If the sensitive values are returned as module outputs, the outputs can be marked sensitive as well
• Terraform 0.15 and higher have a function sensitive() which will mark a value as sensitive, which can be used for values that come from other sources

[caleb15]

Thanks for the quick reply. The cloudformation I'm using has the DdApiKey parameter marked with NoEcho. The DdApiKey input should also be sensitive, as it comes from a ssm_parameter data source and the docs say This value is always marked as sensitive in the Terraform plan output, regardless of type. However, I still see the API key in plaintext.

Before running terraform I manually created a datadog_api_key ssm parameter encrypted with a KMS key. Our terraform code: (we are using terraform v0.14.10 + terragrunt v0.28.19)

```hcl #data.tf data "aws_caller_identity" "current" {} data "aws_ssm_parameter" "datadog_app_key" { name = "/${var.vpc_name}/fifteen5/deploy/datadog_app_key" } data "aws_ssm_parameter" "datadog_api_key" { name = "/${var.vpc_name}/fifteen5/deploy/datadog_api_key" } data "aws_ssm_parameter" "datadog_external_id" { name = "/${var.vpc_name}/fifteen5/deploy/datadog_external_id" } # datadog.tf resource "aws_cloudformation_stack" "datadog" { name = "datadog" capabilities = ["CAPABILITY_AUTO_EXPAND", "CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"] template_url = "https://datadog-cloudformation-template.s3.amazonaws.com/aws/main.yaml" parameters = { ExternalId = data.aws_ssm_parameter.datadog_external_id.value DdApiKey = data.aws_ssm_parameter.datadog_api_key.value LogArchives = "15five-${var.vpc_name}-logs/datadogLogArchives" } tags = { Terraform = "True" } } # main.tf provider "aws" { profile = var.aws_profile region = var.aws_region assume_role { role_arn = var.aws_role } } provider "datadog" { api_key = data.aws_ssm_parameter.datadog_api_key.value app_key = data.aws_ssm_parameter.datadog_app_key.value } # variables.tf ### Variables variable "aws_region" { description = "The AWS region to deploy to (e.g. us-east-1)" type = string default = "us-east-1" } variable "aws_profile" { description = "AWS access profile" type = string } variable "aws_role" { description = "AWS Role to assume access" type = string } variable "environment" { type = string description = "Environment for VPC (eg. production, staging, etc)" } variable "vpc_name" { type = string description = "VPC Name" } variable "vpc_id" { type = string description = "VPC ID" } # versions.tf terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 3.28.0" } datadog = { source = "DataDog/datadog" version = "~> 3.1.2" } } required_version = ">= 0.13" } ```

The plan:

Terraform will perform the following actions:

  # aws_cloudformation_stack.datadog will be updated in-place
  ~ resource "aws_cloudformation_stack" "datadog" {
        id               = "arn:aws:cloudformation:us-east-1:<not shown>:stack/datadog/<not shown>"
        name             = "datadog"
      ~ parameters       = {
          ~ "DdApiKey"    = "****" -> "This should not be shown in plaintext"
            # (2 unchanged elements hidden)
        }
        tags             = {
            "Terraform" = "True"
        }
        # (5 unchanged attributes hidden)
    }

[caleb15]

We just upgraded to terraform v15 and that fixed the issue of sensitive vars being shown in plaintext.

https://www.terraform.io/upgrade-guides/0-15.html

Terraform v0.15 extends this mechanism to also work for values derived from resource attributes that the provider has declared as being sensitive

You can hide my comments - I sidetracked this issue, sorry.

[wesleykirklandsg]

Adding my own comment to this. I was having update issues with a CF stack forever updating and ran across this. I had to a lifecycle action to ignore the parameter input to prevent continual updates.

[ActionScripted]

Same issue here funnily enough also with DdApiKey. For now we're using a lifecycle block as @wesleykirklandsg suggested:

resource "aws_cloudformation_stack" "datadog_forwarder" {
  # SNIP (all the other standard DD stuff here)
  # SNIP (all the other standard DD stuff here)

  # For now, ignore changes to the API key. This is because when Terraform checks for changes AWS/CloudFormation
  # will return it as a sensitive value, and Terraform will not be able to compare it to the value in the state.
  #
  # Open issue that suggested this as a temporary workaround:
  # https://github.com/hashicorp/terraform-provider-aws/issues/10300
  lifecycle {
    ignore_changes = [
      parameters["DdApiKey"],
    ]
  }
}

[tormodmacleod]

we're using a aws_cloudformation_stack_set resource to instrument all of our accounts in newrelic. we run into the same problem as above where we're supplying the newrelic license key as a parameter. every time we run a plan, the diff says that the parameter needs to be updated

we tried implementing something similar to the solution mentioned by @wesleykirklandsg and @ActionScripted as below

  lifecycle {
    ignore_changes = [
      parameters["NewRelicLicenseKey"],
    ]     
  }

this works fine as long as you don't try to instrument any new accounts. if you do, it'll fail to configure the firehose resource properly. the only way to fix it is to remove the lifecycle policy. this behaviour is probably well understood by most but it took me by surprise so i thought i'd leave a comment for anyone looking to do something similar to us