search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.58k stars 4.62k forks source link

Cannot create azurerm_monitor_metric_alert for queue or topic of Microsoft.ServiceBus/namespaces #14084

Closed alexs77 closed 2 years ago

alexs77 commented 2 years ago

Community Note

Terraform (and AzureRM Provider) Version

Terraform v1.0.10

Output of terraform init:

2021-11-08T12:25:33.7274425Z ##[section]Starting: Terraform: init
2021-11-08T12:25:33.7283128Z ==============================================================================
2021-11-08T12:25:33.7283461Z Task         : Terraform
2021-11-08T12:25:33.7283866Z Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
2021-11-08T12:25:33.7284244Z Version      : 2.188.1
2021-11-08T12:25:33.7284480Z Author       : Microsoft Corporation
2021-11-08T12:25:33.7284838Z Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
2021-11-08T12:25:33.7285213Z ==============================================================================
2021-11-08T12:25:33.8800863Z [command]/opt/hostedtoolcache/terraform/1.0.10/x64/terraform init -no-color -input=false -var-file=environments/dev.tfvars -var-file=azure_devops.tfvars -backend-config=storage_account_name=tf -backend-config=container_name=devops-alerting -backend-config=key=dev.tfstate -backend-config=resource_group_name=dev -backend-config=subscription_id=*** -backend-config=tenant_id=*** -backend-config=client_id=*** -backend-config=client_secret=***
2021-11-08T12:25:33.9461078Z 
2021-11-08T12:25:33.9462220Z Initializing the backend...
2021-11-08T12:25:33.9527007Z 
2021-11-08T12:25:33.9528230Z Successfully configured the backend "azurerm"! Terraform will automatically
2021-11-08T12:25:33.9529224Z use this backend unless the backend configuration changes.
2021-11-08T12:25:34.1816580Z 
2021-11-08T12:25:34.1818027Z Initializing provider plugins...
2021-11-08T12:25:34.1826360Z - Finding hashicorp/random versions matching "~> 3.1"...
2021-11-08T12:25:34.2871136Z - Finding hashicorp/azuread versions matching "~> 1.6"...
2021-11-08T12:25:34.3302039Z - Finding hashicorp/azurerm versions matching "~> 2.65"...
2021-11-08T12:25:34.4709921Z - Installing hashicorp/random v3.1.0...
2021-11-08T12:25:34.8091664Z - Installed hashicorp/random v3.1.0 (signed by HashiCorp)
2021-11-08T12:25:34.9038266Z - Installing hashicorp/azuread v1.6.0...
2021-11-08T12:25:35.3818433Z - Installed hashicorp/azuread v1.6.0 (signed by HashiCorp)
2021-11-08T12:25:35.4751519Z - Installing hashicorp/azurerm v2.84.0...
2021-11-08T12:25:37.7247862Z - Installed hashicorp/azurerm v2.84.0 (signed by HashiCorp)
2021-11-08T12:25:37.7248239Z 
2021-11-08T12:25:37.7248611Z Terraform has created a lock file .terraform.lock.hcl to record the provider
2021-11-08T12:25:37.7249062Z selections it made above. Include this file in your version control repository
2021-11-08T12:25:37.7249519Z so that Terraform can guarantee to make the same selections by default when
2021-11-08T12:25:37.7249896Z you run "terraform init" in the future.
2021-11-08T12:25:37.7250073Z 
2021-11-08T12:25:37.7250342Z Terraform has been successfully initialized!
2021-11-08T12:25:37.7250511Z 
2021-11-08T12:25:37.7250842Z You may now begin working with Terraform. Try running "terraform plan" to see
2021-11-08T12:25:37.7251309Z any changes that are required for your infrastructure. All Terraform commands
2021-11-08T12:25:37.7251651Z should now work.
2021-11-08T12:25:37.7251772Z 
2021-11-08T12:25:37.7252098Z If you ever set or change modules or backend configuration for Terraform,
2021-11-08T12:25:37.7252548Z rerun this command to reinitialize your working directory. If you forget, other
2021-11-08T12:25:37.7252967Z commands will detect it and remind you to do so if necessary.
2021-11-08T12:25:37.7373577Z ##[section]Finishing: Terraform: init

Affected Resource(s)

Terraform Configuration Files

locals {
  queue_names = [
    "derivationjobcallback",
    "processingjobstart"
  ]

  topic_names = [
    "eventhub",
    "eventstream",
    "internal"
  ]
}

data "azurerm_servicebus_namespace" "this" {
  name                = var.sb_ns_name
  resource_group_name = var.resource_group_name
}

data "azurerm_servicebus_queue" "these" {
  for_each = toset(local.queue_names)
  name     = each.key

  resource_group_name = data.azurerm_servicebus_namespace.this.resource_group_name
  namespace_name      = data.azurerm_servicebus_namespace.this.name
}

data "azurerm_servicebus_topic" "these" {
  for_each = toset(local.topic_names)
  name     = each.key

  resource_group_name = data.azurerm_servicebus_namespace.this.resource_group_name
  namespace_name      = data.azurerm_servicebus_namespace.this.name
}

locals {
  queues_topics = merge(data.azurerm_servicebus_topic.these, data.azurerm_servicebus_queue.these)
}

resource "azurerm_monitor_metric_alert" "sb_dlq_red" {
  for_each = toset(concat(local.queue_names, local.topic_names))
  scopes = [
    lookup(data.azurerm_servicebus_queue.these, each.key, null) == null ? data.azurerm_servicebus_topic.these[each.key].id : data.azurerm_servicebus_queue.these[each.key].id
  ]

  name                = "DLQ Alert for Topics (Red)"
  resource_group_name = var.resource_group_name
  description         = "Action will be triggered when number of messages in Dead Letter Topic is greater than or equal to 1."
  severity            = 2

  criteria {
    metric_namespace = "Microsoft.ServiceBus/namespaces"

    metric_name = "DeadletteredMessages"
    aggregation = "Average"
    operator    = "GreaterThanOrEqual"
    threshold   = 1
  }

  action {
    action_group_id    = azurerm_monitor_action_group.red.id
  }
}

Debug Output

2021-11-08T12:33:26.7683200Z 
2021-11-08T12:33:26.7689576Z Error: creating or updating Monitor Metric Alert: (Name "DLQ Alert for Topics (Red)" / Resource Group "SharedienDevelopment"): insights.MetricAlertsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="ResourceNotFound" Message="{\"code\":\"BadRequest\",\"message\":\"Detect invalid value: Microsoft.ServiceBus/namespaces for query parameter: 'metricnamespace', the value must be: Microsoft.ServiceBus/namespaces/topics if the query parameter is provided, you can also skip this optional query parameter.\"} Activity ID: 799751b4-cf63-44a1-923e-d17c3c024557."
2021-11-08T12:33:26.7692662Z 
2021-11-08T12:33:26.7694269Z   with azurerm_monitor_metric_alert.sb_dlq_red["advellence.sharedien.internal"],
2021-11-08T12:33:26.7695838Z   on service_bus.tf line 42, in resource "azurerm_monitor_metric_alert" "sb_dlq_red":
2021-11-08T12:33:26.7696793Z   42: resource "azurerm_monitor_metric_alert" "sb_dlq_red" {
2021-11-08T12:33:26.7697171Z

Expected Behaviour

Alert is created.

Actual Behaviour

See above.

Steps to Reproduce

  1. terraform init
  2. terraform apply

Important Factoids

Running in Azure DevOps Release Pipeline.

When I change metric_namespace = "Microsoft.ServiceBus/namespaces" to metric_namespace = "Microsoft.ServiceBus/namespaces/topics" as indicated in the error message, I get this error message:

2021-11-08T11:15:43.6628025Z 
2021-11-08T11:15:43.6630233Z   with azurerm_monitor_metric_alert.sb_dlq_red["advellence.sharedien.derivationjobcallback"],
2021-11-08T11:15:43.6631134Z   on service_bus.tf line 42, in resource "azurerm_monitor_metric_alert" "sb_dlq_red":
2021-11-08T11:15:43.6632050Z   42: resource "azurerm_monitor_metric_alert" "sb_dlq_red" {
2021-11-08T11:15:43.6632390Z 
2021-11-08T11:15:43.6632691Z 
2021-11-08T11:15:43.6652266Z Error: creating or updating Monitor Metric Alert: (Name "DLQ Alert for Topics (Red)" / Resource Group "SharedienDevelopment"): insights.MetricAlertsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="ResourceNotFound" Message="{\"code\":\"BadRequest\",\"message\":\"Microsoft.ServiceBus/namespaces/topics is not a supported platform metric namespace, supported ones are Microsoft.AnalysisServices/servers,Microsoft.Web/staticSites,Microsoft.Web/serverFarms,Microsoft.Web/sites,Microsoft.Web/sites/slots,Microsoft.Web/hostingEnvironments,Microsoft.Web/hostingEnvironments/multiRolePools,Microsoft.Web/hostingEnvironments/workerPools,Microsoft.Web/connections,Microsoft.IoTCentral/IoTApps,Microsoft.ServiceBus/namespaces,Microsoft.Network/virtualNetworks,Microsoft.Network/natGateways,Microsoft.Network/publicIPAddresses,Microsoft.Network/networkInterfaces,Microsoft.Network/privateEndpoints,Microsoft.Network/loadBalancers,Microsoft.Network/networkWatchers/connectionMonitors,Microsoft.Network/virtualNetworkGateways,Microsoft.Network/connections,Microsoft.Network/applicationGateways,Microsoft.Network/dnszones,Microsoft.Network/privateDnsZones,Microsoft.Network/trafficmanagerprofiles,Microsoft.Network/expressRouteCircuits,Microsoft.Network/vpnGateways,Microsoft.Network/p2sVpnGateways,Microsoft.Network/expressRouteGateways,Microsoft.Network/expressRoutePorts,Microsoft.Network/azureFirewalls,Microsoft.Network/privateLinkServices,Microsoft.Network/frontdoors,Microsoft.Network/virtualRouters,Microsoft.HealthcareApis/services,Microsoft.HealthcareApis/workspaces/iotconnectors,Microsoft.DBforMySQL/servers,Microsoft.DBforMySQL/flexibleServers,microsoft.insights/components,microsoft.insights/autoscalesettings,Microsoft.Sql/servers,Microsoft.Sql/servers/databases,Microsoft.Sql/servers/elasticpools,Microsoft.Sql/managedInstances,Microsoft.KeyVault/vaults,Microsoft.Cache/Redis,Microsoft.Cache/redisEnterprise,Microsoft.ContainerRegistry/registries,Microsoft.DocumentDB/databaseAccounts,Microsoft.VMwareCloudSimple/virtualMachines,Microsoft.ContainerInstance/containerGroups,Microsoft.Devices/IotHubs,Microsoft.Devices/ElasticPools,Microsoft.Devices/ElasticPools/IotHubTenants,Microsoft.Devices/ProvisioningServices,Microsoft.Compute/virtualMachines,Microsoft.Compute/virtualMachineScaleSets,Microsoft.Compute/virtualMachineScaleSets/virtualMachines,Microsoft.Compute/cloudServices,Microsoft.Compute/cloudServices/roles,Microsoft.Synapse/workspaces,Microsoft.Synapse/workspaces/bigDataPools,Microsoft.Synapse/workspaces/sqlPools,Microsoft.Synapse/workspaces/kustoPools,Microsoft.ClassicCompute/domainNames/slots/roles,Microsoft.ClassicCompute/virtualMachines,Microsoft.SignalRService/SignalR,Microsoft.SignalRService/WebPubSub,Microsoft.DataBoxEdge/DataBoxEdgeDevices,Microsoft.StorageCache/caches,Microsoft.Search/searchServices,Microsoft.Logic/workflows,Microsoft.Logic/integrationServiceEnvironments,Microsoft.HDInsight/clusters,Microsoft.MachineLearningServices/workspaces,Microsoft.DataCollaboration/workspaces,Microsoft.Purview/accounts,Microsoft.Relay/namespaces,Microsoft.DigitalTwins/digitalTwinsInstances,Microsoft.EventHub/namespaces,Microsoft.EventHub/clusters,Microsoft.Kusto/clusters,Microsoft.OperationalInsights/workspaces,Microsoft.Maps/accounts,Microsoft.DBforMariaDB/servers,Microsoft.TimeSeriesInsights/environments,Microsoft.TimeSeriesInsights/environments/eventsources,Microsoft.AppPlatform/Spring,Microsoft.DBforPostgreSQL/servers,Microsoft.DBforPostgreSQL/serversv2,Microsoft.DBforPostgreSQL/flexibleServers,Microsoft.StreamAnalytics/streamingjobs,Microsoft.Cdn/profiles,Microsoft.Cdn/CdnWebApplicationFirewallPolicies,Microsoft.AVS/privateClouds,Microsoft.NotificationHubs/namespaces/notificationHubs,Microsoft.ApiManagement/service,Microsoft.Storage/storageAccounts,Microsoft.Storage/storageAccounts/blobServices,Microsoft.Storage/storageAccounts/tableServices,Microsoft.Storage/storageAccounts/queueServices,Microsoft.Storage/storageAccounts/fileServices,Microsoft.Media/mediaservices,Microsoft.Media/mediaservices/streamingEndpoints,Microsoft.Media/mediaservices/liveEvents,Microsoft.DataLakeAnalytics/accounts,Microsoft.ServiceFabricMesh/applications,Microsoft.PowerBIDedicated/capacities,Microsoft.ClassicStorage/storageAccounts,Microsoft.ClassicStorage/storageAccounts/blobServices,Microsoft.ClassicStorage/storageAccounts/tableServices,Microsoft.ClassicStorage/storageAccounts/fileServices,Microsoft.ClassicStorage/storageAccounts/queueServices,microsoft.aadiam/azureADMetrics,Microsoft.AppConfiguration/configurationStores,Microsoft.Automation/automationAccounts,Microsoft.DataLakeStore/accounts,Microsoft.DataFactory/dataFactories,Microsoft.DataFactory/factories,Microsoft.NetApp/netAppAccounts/capacityPools,Microsoft.NetApp/netAppAccounts/capacityPools/volumes,Microsoft.DataShare/accounts,Microsoft.Peering/peerings,Microsoft.Peering/peeringServices,Microsoft.StorageSync/storageSyncServices,Microsoft.MixedReality/spatialAnchorsAccounts,Microsoft.MixedReality/remoteRenderingAccounts,Microsoft.ContainerService/managedClusters,Microsoft.Blockchain/blockchainMembers,Microsoft.Batch/batchAccounts,Microsoft.EventGrid/eventSubscriptions,Microsoft.EventGrid/topics,Microsoft.EventGrid/domains,Microsoft.EventGrid/extensionTopics,Microsoft.EventGrid/systemTopics,Microsoft.EventGrid/partnerNamespaces,Microsoft.EventGrid/partnerTopics,Microsoft.CognitiveServices/accounts,Microsoft.Media/videoanalyzers,Microsoft.MachineLearningServices/workspaces/onlineEndpoints,Microsoft.MachineLearningServices/workspaces/onlineEndpoints/deployments,microsoft.keyvault/managedhsms,microsoft.network/vpngateways,microsoft.network/bastionHosts,Microsoft.Network/applicationgateways,microsoft.network/virtualnetworkgateways,Microsoft.Network/virtualHubs,microsoft.network/p2svpngateways,Microsoft.Cloudtest/pools,Microsoft.Cloudtest/hostedpools,microsoft.avs/privateClouds,microsoft.compute/disks,Microsoft.Compute/virtualmachineScaleSets,Microsoft.Logic/IntegrationServiceEnvironments,Microsoft.Logic/Workflows,Microsoft.Cdn/cdnwebapplicationfirewallpolicies,Microsoft.DocumentDB/cassandraClusters,Microsoft.DocumentDB/DatabaseAccounts,Wandisco.Fusion/migrators/liveDataMigrations,Wandisco.Fusion/migrators/metadataMigrations,Wandisco.Fusion/migrators,Microsoft.ConnectedVehicle/platformAccounts,Microsoft.Communication/CommunicationServices,microsoft.hybridnetwork/networkfunctions,microsoft.purview/accounts,Microsoft.Cache/redis,microsoft.kubernetes/connectedClusters,Microsoft.DBForPostgreSQL/serverGroupsv2,Microsoft.RecoveryServices/Vaults. TraceId: {b2c253ec-2738-46d5-85ba-689769569f4b}\"} Activity ID: 86e3074f-b7bf-41ec-96be-f7bdca33944d."

Please note, that "Microsoft.ServiceBus/namespaces/topics" is missing in the list.

I get the same errors when trying to use "Microsoft.ServiceBus/namespaces/queues; this is also not listed.

References

Ramblurr commented 2 years ago

I am receiving the same error but with MySQL database resoruces:

Message="{\"code\":\"BadRequest\",\"message\":\"Detect invalid value: Microsoft.DBforMySQL/servers for query parameter: 'metricnamespace', the value must be: Microsoft.DBforMySQL/servers/databases if the query parameter is provided, you can also skip this optional query parameter.\"}
resource "azurerm_monitor_metric_alert" "db_cpu_anomaly" {
  name                = "${module.label_alert.id}-db-cpu"
  enabled             = true
  resource_group_name = data.azurerm_resource_group.main.name
  scopes              = [azurerm_mysql_database.main.id]
  description         = "XXXX ${var.environment} database cpu usage exceeding the previously estimated average."
  frequency           = "PT15M"
  severity            = var.is_production ? local.severity_informational : local.severity_verbose
  window_size         = "PT15M"

  dynamic_criteria {
    metric_namespace         = "Microsoft.DBforMySQL/servers"
    metric_name              = "cpu_percent"
    aggregation              = "Average"
    operator                 = "GreaterOrLessThan"
    alert_sensitivity        = "Medium"
    evaluation_total_count   = 4
    evaluation_failure_count = 4
  }
  action {
    action_group_id = azurerm_monitor_action_group.main.id
  }
}

The official Microsoft list of metric namespaces does not include Microsoft.DBforMySQL/servers/databases

alexs77 commented 2 years ago

@Ramblurr - that's how I do cpu_percent alerting for MS SQL:

resource "azurerm_monitor_metric_alert" "db_cpu_yellow" {
  name                = "DB CPU Alert (Yellow)"
  resource_group_name = data.azurerm_resource_group.this.name
  description         = "Action will be triggered when database CPU percentage is greater than or equal to ${var.db_cpu_yellow}, but less than ${var.db_cpu_red}."
  scopes              = [data.azurerm_mssql_database.this.id]
  severity            = 2

  criteria {
    metric_namespace = "Microsoft.Sql/servers/databases"
    metric_name      = "cpu_percent"
    aggregation      = "Average"
    operator         = "GreaterThanOrEqual"
    threshold        = var.db_cpu_yellow
  }

  criteria {
    metric_namespace = "Microsoft.Sql/servers/databases"
    metric_name      = "cpu_percent"
    aggregation      = "Average"
    operator         = "LessThan"
    threshold        = var.db_cpu_red
  }

  action {
    action_group_id    = azurerm_monitor_action_group.yellow.id
    webhook_properties = {}
  }
}

Where did you find that Microsoft.DBforMySQL/servers would be a legal value? To find legal values, I do:

# Namespace & Metric Name, see
$ az monitor metrics list-definitions --resource /subscriptions/$subscriptionId/resourceGroups/$rgName/providers/Microsoft.Sql/servers/$serverName/databases/$databaseName
Ramblurr commented 2 years ago

I found the namespaces from this list https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftdbformysqlservers

alexs77 commented 2 years ago

My mistake - you're using MySQL. What I've sent is for MS SQL.

Ramblurr commented 2 years ago

Yea, that page lists Microsoft.Sql/servers/databases for MS SQL, and Microsoft.DBforMySQL/servers for MySql..

Also az monitor metrics list-definitions for my MySQL resource lists Microsoft.DBforMySQL/servers

alexs77 commented 2 years ago

Still an issue with hashicorp/azurerm v2.86.0.

alexs77 commented 2 years ago

Turns out that it was because of a wrong scope. Changing to azurerm_servicebus_namespace made it work.

Here's a fully working example:


provider "azurerm"{
    features{}
}

data "azurerm_resource_group" "this" {
  name = "myrg"
}

resource "azurerm_monitor_action_group" "example" {
  name                = "Error Alerts Action einahwa"
  resource_group_name = data.azurerm_resource_group.this.name
  short_name          = "p4action"

  email_receiver {
    name                    = "sendtodevops"
    email_address           = "devops@contoso.com"
    use_common_alert_schema = true
  }
}

resource "azurerm_servicebus_namespace" "example" {
  name                = "servicebus-namespace-einahwa"
  location            = data.azurerm_resource_group.this.location
  resource_group_name = data.azurerm_resource_group.this.name
  sku                 = "Standard"
}

resource "azurerm_servicebus_queue" "example" {
  name                = "servicebus_queue_einahwa"
  resource_group_name = data.azurerm_resource_group.this.name
  namespace_name      = azurerm_servicebus_namespace.example.name
  enable_partitioning = true
}

resource "azurerm_monitor_metric_alert" "sb_dlq_example" {

  scopes = [ azurerm_servicebus_namespace.example.id ]

  name                = "DLQ Alert einahwa"
  resource_group_name = data.azurerm_resource_group.this.name
  description         = "DLQ Alert."
  severity            = 4

  criteria {
    metric_namespace = "Microsoft.ServiceBus/namespaces"

    metric_name = "DeadletteredMessages"
    aggregation = "Average"
    operator    = "GreaterThanOrEqual"
    threshold   = 5
  }

  action {
    action_group_id    = azurerm_monitor_action_group.example.id
  }
}
github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.