Gateway authentication failed for 'Microsoft.Network for Azure Private DNS Zone data sources

[anttipo]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.1

AzureRM Provider Version

3.7.0

Affected Resource(s)/Data Source(s)

azurerm_private_dns_zone

Terraform Configuration Files

We have a module that digs up the Azure Private DNS Zone resources for about 30 or so zones:

data "azurerm_private_dns_zone" "sql_database" {
  provider            = azurerm.hub
  name                = "privatelink.database.windows.net"
  resource_group_name = "rg-networking"
}

data "azurerm_private_dns_zone" "acr" {
  provider            = azurerm.hub
  name                = "privatelink.azurecr.io"
  resource_group_name = "rg-networking"
}

data "azurerm_private_dns_zone" "blob" {
  provider            = azurerm.hub
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = "rg-networking"

Debug Output/Panic Output

When running terraform plan, random dns zones are being read for 20mins before getting a timeout. These are not consistent, and on the next run it is usually different dns zones that are failing:

│ Error: reading Private Dns Zone (Subscription: "xxxx"
│ Resource Group Name: "nvaq-rg-networking"
│ Private Zone Name: "privatelink.database.windows.net"): privatezones.PrivateZonesClient#Get: Failure responding to request: StatusCode=500 -- Original Error: autorest/azure: Service returned an error. Status=500 Code="GatewayAuthenticationFailed" Message="Gateway authentication failed for 'Microsoft.Network'. Diagnostic information: timestamp '20220601T082831Z', tracking id 'ad62a43f-0488-4419-a3b0-49641df31e01', request correlation id '361a5632-68f6-98ca-f418-c1e7b384c38b'."
│ 
│   with module.dns_ids.data.azurerm_private_dns_zone.sql_database,
│   on /home/vscode/.terraform.cache/modules/dns_ids/modules/dns_ids/private_zones.tf line 1, in data "azurerm_private_dns_zone" "sql_database":
│    1: data "azurerm_private_dns_zone" "sql_database" {

Expected Behaviour

Data sources should be read as expected

Actual Behaviour

No response

Steps to Reproduce

terraform plan

Important Factoids

Similar results with any TF core versions from 1.1 upwards, as well as different azurerm versions.

References

No response

[rcskosir]

Thanks for opening this issue. Have you been able to try this on any of the more recent 3.x versions or on the new 4.x versions to see if this issue still persists?