azurerm_app_configuration fails during replica creation

kbargobdo commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.3.7

AzureRM Provider Version

3.87

Affected Resource(s)/Data Source(s)

azurerm_app_configuration

Terraform Configuration Files

#Module
resource "azurerm_app_configuration" "app_configuration" {
  name                          = "${var.tenant}-${var.region_abbr}-${var.subscription}-${var.ac_name_short}-ac"
  location                      = var.region
  resource_group_name           = var.resource_group_name
  sku                           = "standard"
  public_network_access         = var.ac_public_network_access

  dynamic "replica" {
    for_each = var.ac_replica_enabled == true ? [var.ac_replica_enabled] : []
    content {
      name     = var.ac_replica_name_short
      location = var.ac_replica_region
    }
  }

  lifecycle {
    ignore_changes = [
      # Ignore changes to tags, e.g. because a management agent
      # updates these based on some ruleset managed elsewhere.
      tags,
    ]
  }
}

resource "azurerm_private_endpoint" "app_configuration_pep" {
  count               = var.ac_private_link ? 1 : 0
  depends_on          = [azurerm_app_configuration.app_configuration]
  name                = "${azurerm_app_configuration.app_configuration.name}-pep"
  location            = var.region
  resource_group_name = var.resource_group_name
  subnet_id           = var.sn_id

  private_dns_zone_group {
    name                 = "privatelink.azconfig.io"
    private_dns_zone_ids = [var.ac_private_dns_zone_id]
  }

  private_service_connection {
    is_manual_connection           = "false"
    name                           = "${azurerm_app_configuration.app_configuration.name}-endpoint"
    private_connection_resource_id = azurerm_app_configuration.app_configuration.id
    subresource_names              = ["configurationStores"]
  }

  lifecycle {
    ignore_changes = [
      # Ignore changes to tags, e.g. because a management agent
      # updates these based on some ruleset managed elsewhere.
      tags,
    ]
  }
}

#Resource
module "app_configuration" {
  source = "*****/_git/terraform-azure-appconfiguration?ref=v1.1.0"

  ac_identity_type         = var.ac_identity_type
  ac_name_short            = var.ac_name_short
  ac_private_link          = var.ac_private_link
  ac_public_network_access = var.ac_public_network_access
  ac_replica_enabled       = var.ac_replica_enabled
  ac_replica_name_short    = var.ac_replica_name_short
  ac_replica_region        = var.ac_replica_region
  region                   = var.region
  region_abbr              = var.region_abbr
  resource_group_name      = var.resource_group_name
  sn_id                    = var.sn_id
  subscription             = var.subscription
  tenant                   = var.tenant
}

variable "ac_identity_type" {}
variable "ac_name_short" {}
variable "ac_private_link" {}
variable "ac_public_network_access" {}
variable "ac_replica_enabled" {}
variable "ac_replica_name_short" {}
variable "ac_replica_region" {}
variable "region" {}
variable "region_abbr" {}
variable "resource_group_name" {}
variable "sn_id" {}
variable "subscription" {}
variable "tenant" {}

# Variables
  # App Configuration Parameters
  ac_name_short            = "name"  # Name of the app configuration resource
  ac_identity_type         = "systemassigned"          # systemassigned or userassigned
  ac_public_network_access = "Disabled"                # Public network access to App Configuration - "Enabled" or "Disabled"
  ac_private_link          = true                      # true if private link, false if not (publically accessible)
  ac_replica_enabled       = true                      # If geo-redundant replica is enabled - true or false
  ac_replica_region        = "southcentralus"          # Region of the geo-redundant replica - typically "southcentralus"
  ac_replica_name_short    = "secondary"               # The name of the secondary replica - typically "secondary"

Debug Output/Panic Output

#Terraform
│ Error: creating Replica (Subscription: "******"
│ Resource Group Name: "******"
│ Configuration Store Name: "*******"
│ Replica Name: "secondary"): polling after Create: polling failed: the Azure API returned the following error:
│ 
│ Status: "Failed"
│ Code: "Conflict"
│ Message: "The request for the resource '****' conflicted with an ongoing operation."
│ Activity Id: "78e3ca11-1cfb-4df4-abda-4788991454ae"
│ 
│ ---
│ 
│ API Response:
│ 
│ ----[start]----
│ {"id":"/subscriptions/*****/providers/Microsoft.AppConfiguration/locations/northcentralus/operationsStatus/exqAzcSU3Fto_g5XFTa9afdNcajwKOECS7-gQPySsBQ","name":"exqAzcSU3Fto_g5XFTa9afdNcajwKOECS7-gQPySsBQ","status":"Failed","error":{"code":"Conflict","message":"The request for the resource '*****' conflicted with an ongoing operation.","additionalInfo":[{"type":"ActivityId","info":{"activityId":"78e3ca11-1cfb-4df4-abda-4788991454ae"}}]}}
│ -----[end]-----
│ 
│ 
│   with module.app_configuration.azurerm_app_configuration.app_configuration,
│   on .terraform\modules\app_configuration\main.tf line 1, in resource "azurerm_app_configuration" "app_configuration":
│    1: resource "azurerm_app_configuration" "app_configuration" {
│ 
╵

#Activity Log
Resource
/subscriptions/*****/resourceGroups/*****/providers/Microsoft.AppConfiguration/configurationStores/*****/replicas/secondary
Operation name
Create Replica
Time stamp
Tue Jun 04 2024 15:02:51 GMT-0400 (Eastern Daylight Time)
Event initiated by
*****
Error code
ResourceOperationFailure
Message
The resource operation completed with terminal provisioning state 'Failed'.

Expected Behaviour

App Configuration resource builds as expected, including the replica in the other region if toggled by variable

Actual Behaviour

App Configuration resource builds and then errors out when attempting to create replica.

Note that this was working a few months ago and we have not changed anything within our Azure subscriptions, modules, or versions of terraform / terragrunt / azurerm

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

teowa commented 4 weeks ago

Hi @kbargobdo , thanks for submitting the issue! I tried to reproduce the issue in my local but I can't, could you please provide minimal examples to reproduce it? From the error message The request for the resource '****' conflicted with an ongoing operation., I guess there might be other ongoing operations modifying the same App Configuration.

kbargobdo commented 4 weeks ago

Hi @teowa,

We discovered that the conflict is due to the another process as you mentioned. This other process leverages an eventgrid trigger and function app to tag the app configuration resource once it sees it was created. However, we did not have this issue a few months ago and nothing with that other process has changed. Is it possible that something changed within Microsoft's APIs that are invoked by the creation of this resource?

We are investigating a few alternatives, including turning off the tagging process on app configuration resource type (providers/Microsoft.AppConfiguration/configurationStores/) or leveraging AZAPI to create the replica after a time_sleep. With these options we likely have a path forward, but any information confirming that those APIs did or did not change would be beneficial.

teowa commented 3 weeks ago

Hi @kbargobdo , you can submit an issue at https://github.com/Azure/AppConfiguration/issues, and the official team will help you with the API details.

hashicorp / terraform-provider-azurerm