Entire local-exec command output suppressed with use of sensitive variable

[marshallford]

Marking a variable as sensitive and referencing that variable in a local-exec provisioner command or in the environment block suppresses all lines of the command output.

My use case (not shown in example TF): Ansible playbook called via a null_resource local-exec provisioner where values required by Ansible are passed via environment variables. As you might expect the Ansible output is crucial for troubleshooting issues and watching for timeouts, especially for complex config mgmt situations. I'd rather have the variable not marked as sensitive than lose the command output -- bummer.

Perhaps Terraform could just redact the value itself?

Thank you TF Team!

Terraform Version

Terraform v0.14.0
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.3
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Terraform Configuration Files

variable "foo" {
  type      = string
  sensitive = true
}

# environment variable example
resource "null_resource" "foobar" {
  triggers = {
    always_run = timestamp()
  }

  provisioner "local-exec" {
    command     = "command-that-does-not-include-templated-reference-to-var-foo"
    environment = {
      FOO = var.foo
    }
  }
}

Debug Output

N/A?

Crash Output

N/A

Expected Behavior

As mentioned above, ideally the behavior is similar to CI platforms (Jenkins comes to mind) in that Terraform would only mask/suppress the actual variable value. For example, take the "simple" resource example shown in a blog post on the topic. Even after marking the values as sensitive this Terraform output can still be reasoned about. Unlike lines and lines of output suppressed due to sensitive value in config.

Terraform will perform the following actions:

  # some_resource.a will be created
  + resource "some_resource" "a" {
      + name    = (sensitive) # still digestible 
      + address = (sensitive)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Lastly, one comment that I expect to come up is "a local-exec command might output a value derived from the sensitive variable, so the entire output is suppressed". I guess my response to that would be a feature request 😄, a local-exec option that allows for opt-in "partial protection" if it known that the command output won't contain a derived sensitive value. A best effort of sorts so that the raw value would be redacted (*******), but beyond that all bets are off.

This idea (or the bug report) is not completely baked -- I'm open to a discussion or just having this issue closed.

Actual Behavior

.null_resource.foobar: Provisioning with 'local-exec'...
null_resource.foobar (local-exec): (output suppressed due to sensitive value in config)
null_resource.foobar (local-exec): (output suppressed due to sensitive value in config)
null_resource.foobar (local-exec): (output suppressed due to sensitive value in config)
...

Steps to Reproduce

N/A

Additional Context

N/A

References

• related: https://github.com/hashicorp/terraform/issues/17568

[danieldreier]

@marshallford the behavior you're describing is how we expect local-exec to behave with sensitive output, for now. I'm going to re-label this as an enhancement request, because the idea of redacting output (rather than suppressing it entirely) is a valid one, but was out of scope for our first pass at this, where we felt it more important to hide too much than to risk accidentally exposing secrets.

[marshallford]

Fair enough -- thanks Daniel!

[aeriksson]

Is there a way to work around this? I have a shell script to which I'm passing a sensitive value as an env var, that I'm certain doesn't print said env var, and where I really need to see the output.

[vfuyong]

Same request here. It is not nice to just hide everything. People need to read the output.

[hal58th]

While not fixing OP's issue, I think an "ignore-sensitive" config option would be useful for local/remote-exec. I envision temporarily enabling this for testing purposes so we can see the output of commands. It would be easier to implement in the mean time over searching for certain values and redacting them.

[hal58th]

Release 0.15.0-rc2 has a nonsensitive function and will probably solve OP's issue. This can probably be marked as closed unless we want local-exec output to search for the specific secret to selectively redact them. Documentation of nonsenstive function.

[achevalet]

What's confusing is that we have the clear output when it returns non-zero value:

[Yegair]

What's confusing is that we have the clear output when it returns non-zero value:

Indeed, exposing secrets in case of an error is the real issue here!

I would argue that redacting the first line, which echoes the command that is about to be executed, would be enough.

[alrf]

How to use nonsensitive with null_resource? I don't have variables marked as sensitive = true, however null_resource output is:
(local-exec): (output suppressed due to sensitive value in config)

p.s. fixed. Any values coming from AWS Secret Manager are considered as sensitive, workaround: nonsensitive(data.aws_secretsmanager_secret_version.cluster.secret_string)

[klemmchr]

A workaround for this exact use case (namely running ansible-playbook) is to pass extra vars as a file instead of passing them directly. The YAML/JSON can be created with a local_file resource beforehand.

[jurgen-weber-deltatre]

if you are just running a little bash script, you can wrap the contents of it like so:

{
  echo "my super awesome bash"
} >> my.super.awesome.bash.log

and now we have a log file so we can debug. :)

[dhmoto17]

Is there any movement on a native solution to this? Writing to a log isn't a simple fix for our environment as our terraform runs on a build server. I can think of a few ways to get at an output file but it feels pretty hacky when the previous versions of terraform just output to screen.

[dhmoto17]

The workaround for me is to use the function nonsensitive() around the variable I was passing to my bash script. YMMV if this is suitable to all instances of the problem, depending on how sensitive the value is and how secure the bash script is coded. Example we are using a bash script to apply SQL permissions using sqlcmd.

resource "null_resource" "grant_db_access" {
  for_each = azurerm_sql_database.db

  provisioner "local-exec" {
    command = format("/bin/bash %s/scripts/grant_db_access.sh", var.bash_env.TERRAFORM_DEPLOYMENT)

    environment = {
      SQLCMDUSERNAME = data.terraform_remote_state.config.outputs.sql.server_admin_azure_ad_user.user_name
      SQLCMDPASSWORD = nonsensitive(data.azurerm_key_vault_secret.sqladmin_ad_password.value)
      SQLSERVERFQDN  = azurerm_sql_server.server.fully_qualified_domain_name
    }
  }
}

[Poundex]

I don't think it's feasible for us to have to essentially fork every single 3rd party module and wrap everything in nonsensitive to be able to see what what's going wrong at apply time. If Terraform must have this behaviour of making log files useless (rather than people securing who can see which logs in their CI server) then my vote would be for a global config option as suggested by @hal58th

[gnthibault]

I don't think it's feasible for us to have to essentially fork every single 3rd party module and wrap everything in nonsensitive to be able to see what what's going wrong at apply time. If Terraform must have this behaviour of making log files useless (rather than people securing who can see which logs in their CI server) then my vote would be for a global config option as suggested by @hal58th

100% agree with this, especially when using terraform enterprise, and not being able to get access to the hosting machine, it becomes a nightmare to debug, you never now which variable is sensitive, which one is not. A generic nonsensitive option for null_resource / local-exec is definitely needed.

[kvaidas]

It looks like nonsensitive doesn't work when you're passing an entire object to a template and then accessing its properties. There is just no way to make that work. It either covers everything or errors out due to nonsensitive supposedly being redundant. Even for something as trivial as local_file being that object.

[divomen]

+1

[jok0]

+1

I can see this is still open so assuming that there isn't any work around yet?

I am experiencing this issue when trying to run a packer build within a local-exec provisioner. Any work around suggests would be great. Thanks.

[Kristin0]

Agree. Any work around suggests would be great. Thanks.

[filipmacek]

+1 , would be great to have option for this, hard to debug stuff without seeing logs

[svyas8x8]

+1, still any workaround yet fir this?

[crw]

Thanks for your interest in this issue! This is just a reminder to please avoid "+1" comments, and to use the upvote mechanism (click or add the 👍 emoji to the original post) to indicate your support for this issue. This helps avoid notification spam for issues with high numbers of participants while enabling the maintainers to prioritize issues. Thanks again for the feedback!

[denisvll]

+1

[pfrydids]

Work around involving two local-execs:

resource "null_resource" "migration" {

  # this local-exec will have its output suppressed
  provisioner "local-exec" {
    command = <<EOT
      #!/bin/bash
      rm migration_failed
      set -o pipefail
      ./migrate.sh 2>&1 | tee output.log
      if [ $? -ne 0 ]; then touch migration_failed; fi
    EOT

    environment = {
      PGUSER     = module.postgres.username
      PGHOST     = module.postgres.hostname
      PGDATABASE = module.postgrest.database
      PGPASSWORD = module.postgres.password
    }
  }

  # this local-exec will NOT have its output suppressed
  provisioner "local-exec" {
    command = <<EOT
      #!/bin/bash
      echo "output.log:"
      cat output.log
      if [ -e "migration_failed" ]; then exit 1; fi
    EOT
  }
}

it means that any migration script run will have its output available within the Terraform apply output but also when that migration script has failed the Terraform apply run will fail which is useful when running in any CI/CD pipeline.

The first local-exec should never fail (!) but the second will fail when the migration script has failed.

[ep4sh]

Hi there, is there any chance to fix this issue?

[MAN-Sendance]

in my opinion that's really a problem, I have no idea what is going on while running "terraform apply" due to that issue.

[ep4sh]

I suppose that these "prayers" are not heard... :(

[crw]

There are no updates at this time.

[jsiwrk]

Another workaround: modify your provisioning script to receive the secrets via environment variables, save the secrets to a file, and source the file before executing your script.

  // The output of this provisioner will be suppressed
  provisioner "remote-exec" {
    inline = [
      "echo \"export SECRET_USERNAME='${var.secret_user}'\" > .secrets",
      "echo \"export SECRET_PASSWORD='${var.secret_password}'\" >> .secrets",
      "chmod 400 .secrets"
    ]
  }

  // The output of this provisioner will not be suppressed
  provisioner "remote-exec" {
    inline = [
      "source .secrets",
      "do-something.sh"
    ]
  }

[rybakovanton-metta]

maybe someone can "fix" bin file and that's all? :)