aws_db_instance_role_association doesn't wait for RDS to finish modifying

xposix commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.2 on darwin_arm64

provider registry.terraform.io/hashicorp/aws v4.17.1
provider registry.terraform.io/hashicorp/random v3.3.0

Affected Resource(s)

aws_db_instance_role_association
aws_db_instance

Terraform Configuration Files

resource "aws_db_instance" "general" {
  identifier            = "${var.environment_name}-database"
  instance_class        = var.rds_instance_type
  port                  = 5432
  allocated_storage     = 15
  max_allocated_storage = var.rds_volume_size_max_threshold

  db_subnet_group_name = aws_db_subnet_group.general.id
  vpc_security_group_ids = [
    aws_security_group.rds.id
  ]
  deletion_protection = false
  enabled_cloudwatch_logs_exports = [
    "postgresql",
    "upgrade"
  ]

  engine                              = "postgres"
  engine_version                      = var.rds_engine_version
  iam_database_authentication_enabled = false
  kms_key_id                          = aws_kms_key.general.arn

  maintenance_window = "sat:01:45-sat:02:15"

  monitoring_interval = 60
  monitoring_role_arn = aws_iam_role.rds_monitoring.arn
  multi_az            = var.rds_multi_az_enabled

  parameter_group_name                  = aws_db_parameter_group.general.name
  performance_insights_enabled          = true
  performance_insights_kms_key_id       = aws_kms_key.general.arn
  performance_insights_retention_period = 7

  skip_final_snapshot       = var.environment_type == "live" ? false : true
  final_snapshot_identifier = "${var.environment_name}-database-final-snapshot-${random_string.final_snapshot_identifier.result}"
  backup_retention_period   = 7
  backup_window             = "01:00-01:44"
  copy_tags_to_snapshot     = true
  snapshot_identifier       = var.rds_snapshot_identifier
  storage_encrypted         = true
  storage_type              = "gp2"

  username = "postgres"
  password = random_password.general_rds_master_user.result

  tags = local.project_tags
}

resource "aws_db_instance_role_association" "rds_import" {
  depends_on = [
    aws_iam_role_policy.rds_glue_jobs_s3_bucket_access
  ]
  db_instance_identifier = aws_db_instance.general.id
  feature_name           = "s3Import"
  role_arn               = aws_iam_role.rds.arn
}

resource "aws_iam_role" "rds" {
  name               = "${var.environment_name}-${var.region}-rds-role"
  assume_role_policy = <<EOP
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "rds.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "${data.aws_caller_identity.current.account_id}",
          "aws:SourceArn": "${aws_db_instance.general.arn}"
        }
      }
    }
  ]
}
EOP
  tags               = local.project_tags
}

resource "aws_iam_role_policy" "rds_glue_jobs_s3_bucket_access" {
  name = "${var.environment_name}-${var.region}-glue-jobs-bucket-read-access"
  role = aws_iam_role.rds.id

  policy = <<EOP
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "s3import01",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_s3_bucket.glue.arn}/*"
      ]
    },
    {
      "Sid": "s3import02",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "${aws_s3_bucket.glue.arn}"
      ]
    }
  ]
}
EOP
}

Debug Output

First terraform apply

│ Error: error associating RDS DB Instance (ops-126b-database) IAM Role (arn:aws:iam::373890764051:role/ops-126b-eu-west-2-rds-role): InvalidDBInstanceState: The status for the ops-126b-database DB instance is modifying. The DB instance is not available for s3Import feature.
│   status code: 400, request id: 02b76f62-936c-4f1e-a407-094d3fa13c72
│
│   with aws_db_instance_role_association.rds_import,
│   on iam.tf line 82, in resource "aws_db_instance_role_association" "rds_import":
│   82: resource "aws_db_instance_role_association" "rds_import" {

All the following terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_db_instance_role_association.rds_import will be created
  + resource "aws_db_instance_role_association" "rds_import" {
      + db_instance_identifier = "ops-126b-database"
      + feature_name           = "s3Import"
      + id                     = (known after apply)
      + role_arn               = "arn:aws:iam::11111111111111:role/ops-126b-eu-west-2-rds-role"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
aws_db_instance_role_association.rds_import: Creating...
aws_db_instance_role_association.rds_import: Still creating... [10s elapsed]
aws_db_instance_role_association.rds_import: Still creating... [20s elapsed]
aws_db_instance_role_association.rds_import: Creation complete after 25s [id=ops-126b-database,arn:aws:iam::11111111111111:role/ops-126b-eu-west-2-rds-role]

Warnings:

- Value for undeclared variable
- Value for undeclared variable
- Values for undeclared variables

To see the full warning notes, run Terraform without -compact-warnings.

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Expected Behavior

It should wait for the RDS instance to finish modifying and perform the required changes on it.

Actual Behavior

It tries to associate the role to the instance when its on modifying status and it fails. Then the user needs to retry to make it work.

cloudsurf-digital commented 2 years ago

In case you want to create two aws_db_instance_role_association using the s3Import and s3Export feature it's even more drastically failing. The first one is failing like @xposix describes with:

│ Error: error associating RDS DB Instance (arn:aws:rds:eu-central-1:xxxxxxxx:db:test) IAM Role (arn:aws:iam::xxxxxx:role/s3-import-role): InvalidDBInstanceState: The status for the test DB instance is modifying. The DB instance is not available for s3Import feature.
│   status code: 400, request id: 53724b88-827a-4302-850e-4b14aaf83391
│ 
│   with aws_db_instance_role_association.rds_import,
│   on rds.tf line 41, in resource "aws_db_instance_role_association" "rds_import":
│   41: resource "aws_db_instance_role_association" "rds_import" {
│

and the second fails with

│ Error: error waiting for RDS DB Instance (arn:aws:rds:eu-central-1:xxxxxxx:db:test) IAM Role (arn:aws:iam::xxxxxx:role/s3-export-role) association: couldn't find resource (21 retries)
│ 
│   with aws_db_instance_role_association.rds_export,
│   on rds.tf line 50, in resource "aws_db_instance_role_association" "rds_export":
│   50: resource "aws_db_instance_role_association" "rds_export" {
│

This happening is on

Terraform v1.2.5 on darwin_arm64

provider registry.terraform.io/hashicorp/aws v4.25.0

Same code as @xposix plus

resource "aws_db_instance_role_association" "rds_export" {
  depends_on = [
    aws_iam_role_policy.rds_s3_bucket_export_policy
  ]
  db_instance_identifier = module.postgresql_test.db_instance_arn
  feature_name           = "s3Export"
  role_arn               = aws_iam_role.rds_export.arn
}

arda-yildirim commented 1 year ago

This behavior is also happening for us when trying to associate roles (first s3-import then s3-export) to RDS Aurora Cluster (Serverless V2) with "aws_rds_cluster_role_association" resource. In the first attempt, it's able to add the first role (s3-import) but after then it's failing with:

Error: error creating RDS DB Cluster (XXX) IAM Role (arn:aws:iam::***:role/s3-export-role) Association: InvalidDBClusterStateFault: DB Cluster XXX is not available for IAM role association, disassociation, or updates for ARN: arn:aws:iam::***:role/s3-export-role. Wait for a moment and try again.
    status code: 400, request id: xxx

  on dbs.tf line 100, in resource "aws_rds_cluster_role_association" "test_export_role_association":
 100: resource "aws_rds_cluster_role_association" "test_export_role_association" {

When we try to rerun the workflow, it's failing with (even if it was able to create and attach the s3-import role to the DB):

Error: DBClusterRoleNotFound: Role ARN arn:aws:iam::***:role/s3-import-role cannot be found for DB Cluster: XXX. Verify your role ARN and try again. You might need to include the feature-name parameter.
    status code: 404, request id: xxx

  on .terraform/modules/db/main.tf line 100, in resource "aws_rds_cluster" "standalone_regional":
  100: resource "aws_rds_cluster" "standalone_regional" {

hashicorp / terraform-provider-aws