aws_secretsmanager_secret_version timing issue

syst0m commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.28

provider.aws v2.70.0
provider.random v2.3.0

Affected Resource(s)

aws_secretsmanager_secret
aws_secretsmanager_secret_version

Terraform Configuration Files

resource "aws_secretsmanager_secret" "ume_redshift_password" {
  name = "ume-redshift-password"
}

resource "aws_secretsmanager_secret_version" "ume_redshift_password" {
  secret_id     = aws_secretsmanager_secret.ume_redshift_password.id
  secret_string = random_password.ume_redshift_password.result
}

resource "random_password" "ume_redshift_password" {
  length      = 20
  special     = false
  min_lower   = 1
  min_upper   = 1
  min_numeric = 1
}

data "aws_secretsmanager_secret_version" "ume_redshift_password" {
  secret_id = aws_secretsmanager_secret.ume_redshift_password.id
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

The random_password resource generates a random password, stores in a secretsmanager secret. The data source retrieves the secret_string attribute, and can be used for interpolation in other places of the config (not shown here).

Actual Behavior

The data source fails to retrieve the AWSCURRENT staging version of the secret.

Error: Secrets Manager Secret "arn:aws:secretsmanager:XXXX:secret:ume-redshift-password-uclIsh" Version "AWSCURRENT" not found

The error is gone after applying a 2nd time, and the apply finishes successfully.

Steps to Reproduce

terraform apply

Important Factoids

N/A

References

N/A

Oliniusz commented 3 years ago

I have the similar issue with the following code:

resource "random_password" "rds_admin" {
  length  = 16
  special = false
}

resource "aws_secretsmanager_secret" "rds_admin" {
  provider                = aws.eu-west-1
  name                    = "db_admin"
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "rds_admin" {
  provider      = aws.eu-west-1
  secret_id     = "db_admin"
  secret_string = random_password.rds_admin.result
}

After the first run of terraform apply I get the error:

Error: error putting Secrets Manager Secret value: ResourceNotFoundException: Secrets Manager can't find the specified secret.

The second terraform apply runs as expected. I guess I could try to put some sleep time as a workaround but I haven't tried it yet.

provider.aws: version = "~> 3.10"

gneveu commented 3 years ago

Hello,

Does it help if you replace secret_id = "db_admin" in the aws_secretsmanager_secret_version by secret_id = aws_secretsmanager_secret.rds_admin.id ?

Oliniusz commented 3 years ago

Hi,

I see what you mean.

For now I've used depends on instead and it also seems to work:

resource "aws_secretsmanager_secret_version" "rds_dbadmin" {
  provider      = aws.eu-west-1
  secret_id     = "db_dbadmin"
  secret_string = random_password.rds_dbadmin.result
  depends_on    = [aws_secretsmanager_secret.rds_dbadmin]
}

But I think your secret_id = aws_secretsmanager_secret.rds_admin.id is cleaner and more elegant - I'm changing my code now.

justinretzolk commented 2 years ago

Hi @syst0m 👋 Thank you for reporting this. Given the comments above and that there's been a few Terraform and AWS provider releases between when you filed this and now, can you confirm whether you're still experiencing this?

digihunch commented 2 years ago

I am still experiencing this issue at random.

{"@level":"info","@message":"module.secretmanager.data.aws_secretsmanager_secret_version.creds: Still refreshing... [1m20s elapsed]","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:00.850905Z","hook":{"resource":{"addr":"module.secretmanager.data.aws_secretsmanager_secret_version.creds","module":"module.secretmanager","resource":"data.aws_secretsmanager_secret_version.creds","implied_provider":"aws","resource_type":"aws_secretsmanager_secret_version","resource_name":"creds","resource_key":null},"action":"read","elapsed_seconds":80},"type":"apply_progress"}
{"@level":"info","@message":"module.secretmanager.aws_vpc_endpoint.secmgr: Still creating... [1m20s elapsed]","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:07.160941Z","hook":{"resource":{"addr":"module.secretmanager.aws_vpc_endpoint.secmgr","module":"module.secretmanager","resource":"aws_vpc_endpoint.secmgr","implied_provider":"aws","resource_type":"aws_vpc_endpoint","resource_name":"secmgr","resource_key":null},"action":"create","elapsed_seconds":80},"type":"apply_progress"}
{"@level":"info","@message":"module.secretmanager.aws_vpc_endpoint.secmgr: Creation complete after 1m22s [id=vpce-096cb4d9b732bab91]","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:09.151525Z","hook":{"resource":{"addr":"module.secretmanager.aws_vpc_endpoint.secmgr","module":"module.secretmanager","resource":"aws_vpc_endpoint.secmgr","implied_provider":"aws","resource_type":"aws_vpc_endpoint","resource_name":"secmgr","resource_key":null},"action":"create","id_key":"id","id_value":"vpce-096cb4d9b732bab91","elapsed_seconds":82},"type":"apply_complete"}
{"@level":"error","@message":"Error: Secrets Manager Secret \"arn:aws:secretsmanager:us-east-1:111222333444:secret:hopeful-foxDatabaseCreds-X80WfS\" Version \"AWSCURRENT\" not found","@module":"terraform.ui","@timestamp":"2022-02-24T01:50:09.349632Z","diagnostic":{"severity":"error","summary":"Secrets Manager Secret \"arn:aws:secretsmanager:us-east-1:111222333444:secret:hopeful-foxDatabaseCreds-X80WfS\" Version \"AWSCURRENT\" not found","detail":"","address":"module.secretmanager.data.aws_secretsmanager_secret_version.creds","range":{"filename":"modules/secmgr/data.tf","start":{"line":19,"column":50,"byte":387},"end":{"line":19,"column":51,"byte":388}},"snippet":{"context":"data \"aws_secretsmanager_secret_version\" \"creds\"","code":"data \"aws_secretsmanager_secret_version\" \"creds\" {","start_line":19,"highlight_start_offset":49,"highlight_end_offset":50,"values":[]}},"type":"diagnostic"}

After cleaning up and retry, then everything works again.

justinretzolk commented 2 years ago

Hey @digihunch 👋 Can you confirm what version of Terraform and the AWS Provider you're using?

digihunch commented 2 years ago

I was on terraform v1.0.11, with AWS provider 3.12.0 I'll try the latest version of both

girvenj commented 2 years ago

I am seeing the original issue as well:

Terraform v1.2.4

AWS provider: 4.24.0

justinretzolk commented 2 years ago

For those still experiencing this issue: The original issue here seems to have been a result of the data.aws_secretsmanager_secret_version being read prior to the aws_secretsmanager_secret_version resource creation.

This can likely be resolved by adding a depends_on block to set up an explicit dependency on the aws_secretsmanager_secret_version resource. Alternatively, data.aws_secretsmanager_secret_version's secret_id could be set to aws_secretsmanager_secret_version.<name>.secret_id to create an implicit dependency.

If you're still experiencing this issue, can you test this to confirm that it resolves your issue?

jamiegosling commented 1 year ago

This still seems to be present in Terraform 1.2.7 with AWS provider version 4.43.0. I have an code block like the following:

resource "random_string" "dbpass" {
  length  = 22
  upper   = true
  lower   = true
  numeric  = true
  special = false
}

locals {
  masterpasswd = random_string.dbpass.result
}

resource "aws_secretsmanager_secret" "password" {
  name = "name"
  force_overwrite_replica_secret = true
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "password" {
  secret_id     = aws_secretsmanager_secret.masterpasswd.id
  secret_string = local.masterpasswd

  depends_on = [
    aws_secretsmanager_secret.masterpasswd
  ]
}

which fails on the first apply

Error: Secrets Manager Secret "arn:aws:secretsmanager:eu-west-2:xxx:secret:xxxx-zIQt0L" Version "AWSCURRENT" not found

and works on the subsequent one.

mims92 commented 1 year ago

Still occurring with Terraform 1.3.9 and AWS provider 4.55

uridium commented 1 year ago

Same with:

Terraform v1.4.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.65.0

justinretzolk commented 1 year ago

Given that this was initially reported on a much older version of the AWS provider, can someone who has run into this issue more recently provider a sample Terraform configuration for us to review?

YusDyr commented 1 year ago

I had this error with that configuration:

test.tfvars

target_endpoint = {
  "dst-all" = {
    engine_name = "docdb"
    port        = 27017
    server_name = "mongodb+srv://latam-tech-dev.btrq7.mongodb.net"
    secret_arn  = "arn:aws:secretsmanager:us-east-1:***:secret:/latam-tech/mongo-atlas/env/dev/credentials/admin/latam-tech-dev-jhRNCn"
    ssl_mode    = "none"
  }
}

main.tf

locals {
  targets_secret_arn = {
    for k, v in var.target_endpoint : k => v.secret_arn
    if contains(keys(v), "secret_arn")
  }
  targets_secret = { for k, v in data.aws_secretsmanager_secret_version.targets_endpoint : k => v.secret_string }
}

variable "target_endpoint" {
  description = "Map of objects that define target endpoint to be created, refer to https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dms_endpoint"
  type        = any
}

data "aws_secretsmanager_secret_version" "targets_endpoint" {
  for_each  = local.targets_secret_arn
  secret_id = each.value
}

Result: terraform apply -var-file=test.tfvars

data.aws_secretsmanager_secret_version.targets_endpoint["dst-all"]: Reading...
╷
│ Error: Secrets Manager Secret "arn:aws:secretsmanager:us-east-1:***:secret:/latam-tech/mongo-atlas/env/dev/credentials/admin/latam-tech-dev-jhRNCn" Version "AWSCURRENT" not found
│ 
│   with data.aws_secretsmanager_secret_version.targets_endpoint["dst-all"],
│   on 1.tf line 14, in data "aws_secretsmanager_secret_version" "targets_endpoint":
│   14: data "aws_secretsmanager_secret_version" "targets_endpoint" {
│ 
╵

Versions:

$ tf version
Terraform v1.2.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v5.15.0

The same story with terraform version 1.4.5

Remark: I replaced aws account id with "***"

YusDyr commented 1 year ago

I found the root cause. AWS_REGION was set to us-west-2 in my case, while used secret arn was for us-east-1!

hashicorp / terraform-provider-aws