TF provider fails to create postgres replica in different location

[okoethibm]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

terraform: 1.3.7 IBM provider: 1.52.1

Affected Resource(s)

• ibm_databases

Terraform Configuration Files

resource "ibm_database" "postgresql_hive" {
  count = local.use_existing_pg ? 0 : 1

  name              = "${var.environment_name}-hive-metastore-pg"
  service           = "databases-for-postgresql"
  plan              = "standard"
  resource_group_id = var.resource_group_id
  location          = "us-south"
  service_endpoints = "public-and-private" 

  key_protect_key           = var.byok_keys["databases.postgresql-hive"].crn
  backup_encryption_key_crn = var.byok_keys["databases.postgresql-hive"].crn

  group {
    group_id = "member"
    memory {
      allocation_mb = var.postgres_memory_master
    }
    disk {
      allocation_mb = var.postgres_disk_master
    }
  }

  timeouts {
    create = "120m" # Extending provisioning time to 120 minutes
  }
}

resource "ibm_database" "postgresql_hive_replica" {
  count = local.use_existing_pg ? 0 : 1

  name              = "${var.environment_name}-hive-metastore-pg-replica"
  service           = "databases-for-postgresql"
  plan              = "standard"
  resource_group_id = var.resource_group_id
  location          = "us-east"  # no problem if us-south is given here
  service_endpoints = "public-and-private"

  remote_leader_id  = ibm_database.postgresql_hive[0].resource_crn

  key_protect_key           = var.byok_keys["databases.postgresql-hive"].crn
  backup_encryption_key_crn = var.byok_keys["databases.postgresql-hive"].crn

  group {
    group_id = "member"
    memory {
      allocation_mb = var.postgres_memory_replica
    }
    disk {
      allocation_mb = var.postgres_disk_replica
    }
  }

  timeouts {
    create = "120m" # Extending provisioning time to 120 minutes
  }
}

Expected Behavior

Should create a postgres database and replica

Actual Behavior

Databases get created but terraform fails with

module.databases.ibm_database.postgresql_hive_replica[0]: Still creating... [10m10s elapsed]
╷
│ Error: [ERROR] Error getting database autoscaling groups: Not Found
│
│   with module.databases.ibm_database.postgresql_hive_replica[0],
│   on ../../modules/databases-setup-module/postgresql-hive.tf line 65, in resource "ibm_database" "postgresql_hive_replica":
│   65: resource "ibm_database" "postgresql_hive_replica" {

The replica database is created successfully and looks okay in the cloud UI, but the terraform state for the database is tainted and any plan on the state (even a plan that would destroy the resource again) fails with

│ Error: [ERROR] Error getting database autoscaling groups: Not Found

So the only way to recover is to explicitly remove the replica database from the state

Steps to Reproduce

Apply terraform config given above

Important Factoids

The config works fine if both master and replica database are in the same location (us-south). The error occurs only when the replica database is created in us-east

[okoethibm]

Plan for replica database creation

  # module.databases.ibm_database.postgresql_hive_replica[0] will be created
  + resource "ibm_database" "postgresql_hive_replica" {
      + adminuser                    = (known after apply)
      + backup_encryption_key_crn    = "crn:v1:bluemix:public:kms:us-south:a/2efba25396b342119f5c750152e00db3:40aeb1aa-f3e1-4a96-94ec-25290541bebb:key:8c4b75f7-6f25-4734-9356-87cde0d66462"
      + configuration_schema         = (known after apply)
      + connectionstrings            = (known after apply)
      + groups                       = (known after apply)
      + guid                         = (known after apply)
      + id                           = (known after apply)
      + key_protect_key              = "crn:v1:bluemix:public:kms:us-south:a/2efba25396b342119f5c750152e00db3:40aeb1aa-f3e1-4a96-94ec-25290541bebb:key:8c4b75f7-6f25-4734-9356-87cde0d66462"
      + location                     = "us-east"
      + members_cpu_allocation_count = (known after apply)
      + members_disk_allocation_mb   = (known after apply)
      + members_memory_allocation_mb = (known after apply)
      + name                         = "dev-us-south-hive-metastore-pg-replica"
      + node_count                   = (known after apply)
      + node_cpu_allocation_count    = (known after apply)
      + node_disk_allocation_mb      = (known after apply)
      + node_memory_allocation_mb    = (known after apply)
      + plan                         = "standard"
      + remote_leader_id             = "crn:v1:bluemix:public:databases-for-postgresql:us-south:a/2efba25396b342119f5c750152e00db3:969b72cb-3e19-4ee2-81a6-8714347c48a7::"
      + resource_controller_url      = (known after apply)
      + resource_crn                 = (known after apply)
      + resource_group_id            = "0bfe3de5106a47ec9081abf40b5fdcb3"
      + resource_group_name          = (known after apply)
      + resource_name                = (known after apply)
      + resource_status              = (known after apply)
      + service                      = "databases-for-postgresql"
      + service_endpoints            = "public-and-private"
      + status                       = (known after apply)
      + tags                         = (known after apply)
      + version                      = (known after apply)

      + auto_scaling {
          + cpu {
              + rate_increase_percent       = (known after apply)
              + rate_limit_count_per_member = (known after apply)
              + rate_period_seconds         = (known after apply)
              + rate_units                  = (known after apply)
            }

          + disk {
              + capacity_enabled             = (known after apply)
              + free_space_less_than_percent = (known after apply)
              + io_above_percent             = (known after apply)
              + io_enabled                   = (known after apply)
              + io_over_period               = (known after apply)
              + rate_increase_percent        = (known after apply)
              + rate_limit_mb_per_member     = (known after apply)
              + rate_period_seconds          = (known after apply)
              + rate_units                   = (known after apply)
            }

          + memory {
              + io_above_percent         = (known after apply)
              + io_enabled               = (known after apply)
              + io_over_period           = (known after apply)
              + rate_increase_percent    = (known after apply)
              + rate_limit_mb_per_member = (known after apply)
              + rate_period_seconds      = (known after apply)
              + rate_units               = (known after apply)
            }
        }

      + group {
          + group_id = "member"

          + disk {
              + allocation_mb = 15360
            }

          + memory {
              + allocation_mb = 3072
            }
        }

      + timeouts {
          + create = "120m"
        }
    }

State of the failed resource

# module.databases.ibm_database.postgresql_hive_replica[0]: (tainted)
resource "ibm_database" "postgresql_hive_replica" {
    adminuser                    = "admin"
    backup_encryption_key_crn    = "crn:v1:bluemix:public:kms:us-south:a/2efba25396b342119f5c750152e00db3:40aeb1aa-f3e1-4a96-94ec-25290541bebb:key:8c4b75f7-6f25-4734-9356-87cde0d66462"
    groups                       = [
        {
            count    = 1
            cpu      = [
                {
                    allocation_count = 0
                    can_scale_down   = true
                    is_adjustable    = true
                    minimum_count    = 3
                    step_size_count  = 1
                    units            = "count"
                },
            ]
            disk     = [
                {
                    allocation_mb  = 15360
                    can_scale_down = false
                    is_adjustable  = true
                    minimum_mb     = 15360
                    step_size_mb   = 512
                    units          = "mb"
                },
            ]
            group_id = "member"
            memory   = [
                {
                    allocation_mb  = 3072
                    can_scale_down = true
                    is_adjustable  = true
                    minimum_mb     = 1024
                    step_size_mb   = 128
                    units          = "mb"
                },
            ]
        },
    ]
    guid                         = "cbcae272-ac59-43bc-90fc-0285e5a58c9f"
    id                           = "crn:v1:bluemix:public:databases-for-postgresql:us-east:a/2efba25396b342119f5c750152e00db3:cbcae272-ac59-43bc-90fc-0285e5a58c9f::"
    key_protect_key              = "crn:v1:bluemix:public:kms:us-south:a/2efba25396b342119f5c750152e00db3:40aeb1aa-f3e1-4a96-94ec-25290541bebb:key:8c4b75f7-6f25-4734-9356-87cde0d66462"
    location                     = "us-east"
    members_cpu_allocation_count = 0
    members_disk_allocation_mb   = 15360
    members_memory_allocation_mb = 3072
    name                         = "dev-us-south-hive-metastore-pg-replica"
    node_count                   = 1
    node_cpu_allocation_count    = 0
    node_disk_allocation_mb      = 15360
    node_memory_allocation_mb    = 3072
    plan                         = "standard"
    remote_leader_id             = "crn:v1:bluemix:public:databases-for-postgresql:us-south:a/2efba25396b342119f5c750152e00db3:969b72cb-3e19-4ee2-81a6-8714347c48a7::"
    resource_controller_url      = "https://cloud.ibm.com/services/crn%3Av1%3Abluemix%3Apublic%3Adatabases-for-postgresql%3Aus-east%3Aa%2F2efba25396b342119f5c750152e00db3%3Acbcae272-ac59-43bc-90fc-0285e5a58c9f%3A%3A"
    resource_crn                 = "crn:v1:bluemix:public:databases-for-postgresql:us-east:a/2efba25396b342119f5c750152e00db3:cbcae272-ac59-43bc-90fc-0285e5a58c9f::"
    resource_group_id            = "0bfe3de5106a47ec9081abf40b5fdcb3"
    resource_group_name          = "crn:v1:bluemix:public:resource-controller::a/2efba25396b342119f5c750152e00db3::resource-group:0bfe3de5106a47ec9081abf40b5fdcb3"
    resource_name                = "dev-us-south-hive-metastore-pg-replica"
    resource_status              = "active"
    service                      = "databases-for-postgresql"
    service_endpoints            = "public-and-private"
    status                       = "active"
    tags                         = []
    version                      = "14"

    group {
        group_id = "member"

        disk {
            allocation_mb = 15360
        }

        memory {
            allocation_mb = 3072
        }
    }

    timeouts {
        create = "120m"
    }
}