azurerm_monitor_diagnostic_setting fails to create with 409 nil error

[chintavasisht]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

0.15.5

AzureRM Provider Version

3.49.0

Affected Resource(s)/Data Source(s)

azurerm_monitor_diagnostic_setting

Terraform Configuration Files

resource "azurerm_monitor_diagnostic_setting" "masterdb" {
  for_each = { for sqldb in var.sqldb_attributes : sqldb.sqldb_name => sqldb }

  name               = "sql-single-${each.key}-master-diagnostic"
  target_resource_id = "<azurerm_mssql_server.mssql.id>/databases/master"

  eventhub_authorization_rule_id = "<event hub authorization rule id>"
  eventhub_name                  = "<event hub nampespace>"

  dynamic "log" {
    for_each = data.azurerm_monitor_diagnostic_categories.lookup[each.key].logs
    content {
      category = log.value
      enabled  = log.value == "SQLSecurityAuditEvents" ? true : false
      retention_policy {
        enabled = false
      }
    }
  }
  dynamic "metric" {
    for_each = data.azurerm_monitor_diagnostic_categories.lookup[each.key].metrics
    content {
      category = metric.value
      enabled  = false
      retention_policy {
        enabled = false
      }
    }
  }
}```

Debug Output/Panic Output

│ Error: creating Monitor Diagnostics Setting "sql-single-sqlsinglebc-master-diagnostic" for Resource "/subscriptions/***/resourceGroups/***/providers/Microsoft.Sql/servers/***/databases/master": diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>
with module.module-sql-single.azurerm_monitor_diagnostic_setting.imperva-masterdb["sqlsinglebc"],on ../observability.tf line 62, in resource "azurerm_monitor_diagnostic_setting" "imperva-masterdb": 62: resource "azurerm_monitor_diagnostic_setting" "imperva-masterdb" {

### Expected Behaviour

Running ```Terraform apply``` should have created the diagnostic setting on any given number of Azure MS SQL DB server via the ```for_each``` loop. The intention is to create the ```azurerm_monitor_diagnostic_setting``` on the masterdb of the SQL server(s) (notice the ```target_resource_id```) in the configuration files. 

### Actual Behaviour

```Terraform apply``` does not consistently succeed. It is a hit or a miss with the apply sometimes running successfully sometimes but failing with the above error several times

### Steps to Reproduce

I have the following created ahead of time:

- 1 Azure SQL server and 3 DBs with the following DTU SKUs: ```Basic```, ```S0```, ```P1```
- Eventhub, event hub namespace and an event hub authorization rule

To reproduce, deploy the ```azurerm_monitor_diagnostic_setting``` on all 3 databases

### Important Factoids

_No response_

### References

_No response_

[chintavasisht]

This needs to be marked/categorized as a bug. Not sure why the bot removed removed the label

[teowa]

Hi @chintavasisht , thanks for submitting this issue. The error message "Failure sending request: StatusCode=409" should be error message returned from Service REST API, indicates that there is a conflict with the current state of the resource. This error can occur if there is an existing diagnostic setting with the same log category as the one you are trying to create. I can reproduce a similar error with below config. Could you please try using different log category for each of the diagnostic setting?

Thanks.

config detail

```hcl resource "azurerm_resource_group" "example" { name = "example-resources" location = "West Europe" } resource "azurerm_mssql_server" "example" { name = "example-sqlservertst" resource_group_name = azurerm_resource_group.example.name location = azurerm_resource_group.example.location version = "12.0" administrator_login = "4dm1n157r470r" administrator_login_password = "4-v3ry-53cr37-p455w0rd" } resource "azurerm_mssql_database" "example" { name = "accexample-db-d" server_id = azurerm_mssql_server.example.id sku_name = "Basic" tags = { foo = "bar" } } resource "azurerm_eventhub_namespace" "example" { name = "accexample-EHN-1231" location = azurerm_resource_group.example.location resource_group_name = azurerm_resource_group.example.name sku = "Basic" } resource "azurerm_eventhub" "example" { name = "accexample-EH" namespace_name = azurerm_eventhub_namespace.example.name resource_group_name = azurerm_resource_group.example.name partition_count = 2 message_retention = 1 } resource "azurerm_eventhub_namespace_authorization_rule" "example" { name = "example" namespace_name = azurerm_eventhub_namespace.example.name resource_group_name = azurerm_resource_group.example.name listen = true send = true manage = true } resource "azurerm_monitor_diagnostic_setting" "example" { name = "example" target_resource_id = azurerm_mssql_database.example.id eventhub_authorization_rule_id = azurerm_eventhub_namespace_authorization_rule.example.id eventhub_name = azurerm_eventhub.example.name enabled_log { category = "SQLSecurityAuditEvents" retention_policy { enabled = false } } metric { category = "AllMetrics" retention_policy { enabled = false } } } resource "azurerm_monitor_diagnostic_setting" "example2" { name = "example2" target_resource_id = azurerm_mssql_database.example.id eventhub_authorization_rule_id = azurerm_eventhub_namespace_authorization_rule.example.id eventhub_name = azurerm_eventhub.example.name enabled_log { category = "SQLSecurityAuditEvents" retention_policy { enabled = false } } metric { category = "AllMetrics" retention_policy { enabled = false } } depends_on = [azurerm_monitor_diagnostic_setting.example] } ```

│ Error: creating Monitor Diagnostics Setting "example2" for Resource "/subscriptions/xxx/resourceGroups/wt-example-resources/providers/Microsoft.Sql/servers/example-sqlserverwt/databases/accexample-db-d": diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an
error. Status=409 Code="Conflict" Message="Data sink '/subscriptions/xxx/resourceGroups/wt-example-resources/providers/Microsoft.EventHub/namespaces/accexample-EHN-1231/authorizationRules/example' is already used in diagnostic setting 'example' for category 'SQLSecurityAuditEvents'. Data sinks can't be reused in different settings on the same category for the same resource."
│
│   with azurerm_monitor_diagnostic_setting.example2,
│   on main.tf line 72, in resource "azurerm_monitor_diagnostic_setting" "example2":
│   72: resource "azurerm_monitor_diagnostic_setting" "example2" {
│

[chintavasisht]

Hi @teowa, Thanks and I was aware that it was due to a conflict. I verified this a few hours later in the Azure portal and I saw the exact error you posted in the Azure portal and realized what was happening. However, what bums me out is the fact that it does succeed sometimes with the same category. Could you try this out with multiple databases with different SKUs and see how it goes?

Im just wondering, how you in your case the error message returns with full detail of the conflict and for me it returns Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>. Could it be inconsistency in how the Azure API is handling it?

[alexander-rondon]

I was having the same issue, Error: updating Monitor Diagnostics Setting ... diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil> But then I found the root cause: Data sink *** is already used in diagnostic setting 'eventhub' for category 'ApplicationGatewayAccessLog'. Data sinks can't be reused in different settings on the same category for the same resource.

Thanks @teowa for pointing in the right direction.

[KenFujimoto12]

I have same problem.

Unable to create new with terraform "azurerm" if azurerm_monitor_diagnostic_setting already exists.

I agree @teowa opinions, but It possible that I create new "azurerm_monitor_diagnostic_setting" having the same log category on Azure Web Portal. So, I guess the phenomenon should be solved.

My code is below↓

resource "azurerm_monitor_diagnostic_setting" "this" {
  name               = "xxxx-activity-log-to-admin-storage-blob"
  target_resource_id = "/subscriptions/${var.subscription_id}"
  storage_account_id = "/subscriptions/${var.admin_subscription_id}/resourceGroups/admin-resource-group/providers/Microsoft.Storage/storageAccounts/xxxxxxxxxx"

  dynamic "enabled_log" {
    for_each = ["Administrative", "Security", "ServiceHealth", "Alert", "Recommendation", "Policy", "Autoscale", "ResourceHealth"]
    content {
      category = enabled_log.value
      retention_policy {
        enabled = true
        days    = 0
      }
    }
  }
}

Error

Error: creating Monitor Diagnostics Setting "xxxx-log-to-admin-storage-blob" for Resource "/subscriptions/xxxxxxxxxx": diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>