azurerm_postgresql_server scaling with replica in different region results in error

NillsF commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.9
+ provider registry.terraform.io/hashicorp/azurerm v2.55.0

Your version of Terraform is out of date! The latest version
is 0.14.10. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

azurerm_postgresql_server

Terraform Configuration Files

Two files:

postgres.tf

resource "azurerm_resource_group" "rg" {
  name     = var.resource_group
  location = var.main_location
}

resource "azurerm_postgresql_server" "postgres_master" {
  name                             = var.main_db_name
  location                         = var.main_location
  resource_group_name              = azurerm_resource_group.rg.name
  sku_name                         = var.sku_name
  administrator_login              = var.administrator_login
  administrator_login_password     = var.administrator_login_password
  version                          = var.postgres_version
  ssl_enforcement_enabled          = true
  ssl_minimal_tls_version_enforced = "TLS1_2"
  backup_retention_days            = var.retention_days
  geo_redundant_backup_enabled     = var.replicas_count != 0 ? true : false
  storage_mb                       = var.storage_mb
  public_network_access_enabled    = var.public_access_enabled
  auto_grow_enabled                = true

  lifecycle {
    ignore_changes = [
      # Autogrow is enabled
      storage_mb,
    ]
  }

}

resource "azurerm_postgresql_server" "postgres_standby" {
  count                            = var.replicas_count
  name                             = "${azurerm_postgresql_server.postgres_master.name}-r-${var.replicas_count}"
  location                         = var.replicas_location
  resource_group_name              = var.resource_group
  sku_name                         = var.sku_name
  version                          = var.postgres_version
  ssl_enforcement_enabled          = true
  ssl_minimal_tls_version_enforced = "TLS1_2"
  storage_mb                       = var.storage_mb
  public_network_access_enabled    = var.public_access_enabled
  create_mode                      = "Replica"
  creation_source_server_id        = azurerm_postgresql_server.postgres_master.id
  auto_grow_enabled                = true

  lifecycle {
    ignore_changes = [
      # Autogrow is enabled
      storage_mb,
    ]
  }

}

var.tf

provider "azurerm" {
  features {}
}

variable "main_db_name" {
    default = "nfpgtst"
}

variable "main_location" {
    default = "westus2"
}
variable "resource_group" {
    default = "pg-test-diff-reg"
}
variable "sku_name" {
    default = "GP_Gen5_4"
}
variable "administrator_login" {
    default = "nilfranadmin"
}
variable "administrator_login_password" {
    default = "superSecure123$"
}
variable "postgres_version" {
    default = 9.6
}
variable "retention_days" {
    default = 7
}
variable "replicas_count" {
    default = 1
}
variable "storage_mb" {
    default = 5120
}
variable "replicas_location" {
    default = "eastus2"
}
variable "public_access_enabled" {
    default = false
    }

Debug Output

https://gist.github.com/NillsF/cda78b2bb2a7cd90ba833bcced9665ae

Panic Output

Expected Behaviour

Scale up should not result in an error.
Actual Behaviour

When scaling up, once the replica gets scaled, an error is emitted:

Error: updating PostgreSQL Server "nfpgtst-r-1" (Resource Group "pg-test-diff-reg"): postgresql.ServersClient#Update: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="ServiceBusy" Message="Service is temporarily busy and the operation cannot be performed. Please try again later."

This can be resolved by running terraform plan and terraform apply a second time.

Steps to Reproduce

In var.tf, set the following sku_name:

variable "sku_name" {
default = "GP_Gen5_4"
}

terraform apply

In var.tf, update sku_name to:

variable "sku_name" {
default = "GP_Gen5_8"
}

terraform apply will emit the error (if replica is in another region)

Important Factoids

References

@aristosvo did some great work implementing locking in the following issue/PR:

10754
10284

aristosvo commented 3 years ago

This is easily reproduced by introducing a secondary location for one of the replicas in Acceptance Test TestAccPostgreSQLServer_scaleReplicas:

❯ make acctests SERVICE='postgres' TESTARGS='-run=Replicas'
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
TF_ACC=1 go test -v ./azurerm/internal/services/postgres -run=Replicas -timeout 180m -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
2021/04/24 15:29:08 [DEBUG] not using binary driver name, it's no longer needed
2021/04/24 15:29:09 [DEBUG] not using binary driver name, it's no longer needed
=== RUN   TestAccPostgreSQLServer_scaleReplicas
=== PAUSE TestAccPostgreSQLServer_scaleReplicas
=== CONT  TestAccPostgreSQLServer_scaleReplicas
    testing.go:620: Step 3/6 error: Error running apply: exit status 1

        Error: updating PostgreSQL Server "acctest-psql-server-210424152910412500-replica2" (Resource Group "acctestRG-psql-210424152910412500"): postgresql.ServersClient#Update: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="ServiceBusy" Message="Service is temporarily busy and the operation cannot be performed. Please try again later."

          on terraform_plugin_test.tf line 43, in resource "azurerm_postgresql_server" "replica2":
          43: resource "azurerm_postgresql_server" "replica2" {

--- FAIL: TestAccPostgreSQLServer_scaleReplicas (1481.86s)
FAIL
FAIL    github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/postgres  1485.338s
FAIL
make: *** [acctests] Error 1

I'll check the options to solve it, probably something like removing sku_name for a secondary.

ghost commented 3 years ago

This has been released in version 2.57.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.57.0"
}
# ... other configuration ...

github-actions[bot] commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm