V4.9.0 ~> management.azure.com dial tcp 4.150.240.10:443: connect: connection timed out

sander-verhoeven commented 1 day ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.9.8

AzureRM Provider Version

4.9.0

Affected Resource(s)/Data Source(s)

azurerm_servicebus_queue

Terraform Configuration Files

Module/servicebus: 

resource "azurerm_servicebus_queue" "queue" {
  name         = local.servicebusQueueName
  namespace_id = local.servicebusQueue.config.servicebusNamespaceId

  dead_lettering_on_message_expiration    = try(local.servicebusQueue.config.deadLetteringOnMessageExpiration, false)
  default_message_ttl                     = try(local.servicebusQueue.config.defaultMessageTtl, "PT20M")
  duplicate_detection_history_time_window = try(local.servicebusQueue.config.duplicateDetectionHistoryTimeWindow, "PT10M")
  batched_operations_enabled              = try(local.servicebusQueue.config.batchedOperationsEnabled, true)
  express_enabled                         = try(local.servicebusQueue.config.expressEnabled, false)
  partitioning_enabled                    = try(local.servicebusQueue.config.partitioningEnabled, false)
  forward_dead_lettered_messages_to       = try(local.servicebusQueue.config.forwardDeadLetteredMessagesTo, null)
  forward_to                              = try(local.servicebusQueue.config.forwardTo, null)
  lock_duration                           = try(local.servicebusQueue.config.lockDuration, "PT1M")
  max_delivery_count                      = try(local.servicebusQueue.config.maxDeliveryCount, 10)
  max_size_in_megabytes                   = try(local.servicebusQueue.config.maxSizeInMegabytes, 1024)
  requires_duplicate_detection            = try(local.servicebusQueue.config.requiresDuplicateDetection, false)
  requires_session                        = try(local.servicebusQueue.config.requiresSession, false)
  status                                  = try(local.servicebusQueue.config.status, "Active")
}

tfvars:

servicebusQueue = [
      {
        servicebusQueueId                    = "01"
        servicebusNamespaceName              = "xxx"
        servicebusNamespaceResourceGroupName = "rg-xx"
        name                                 = "xx"
        subscription = {
          topicName = "xx"
        }
        authorization = {
          sasRole     = []
          adGroupRole = {}
          servicePrincipalRole = {}
        }
      },

Debug Output/Panic Output

2024-11-19T08:55:01.0589335Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: [DEBUG] AzureRM Request: 
2024-11-19T08:55:01.0589884Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: GET /subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview HTTP/1.1
2024-11-19T08:55:01.0590276Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: Host: management.azure.com
2024-11-19T08:55:01.0590840Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 queues/2021-06-01-preview) HashiCorp Terraform/1.9.8 (+https://www.terraform.io) terraform-provider-azurerm/4.10.0+4.0-beta VSTS_xxx_build_1103_0 pid-xxxx
2024-11-19T08:55:01.0591276Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: Content-Type: application/json; charset=utf-8
2024-11-19T08:55:01.0591570Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: Accept-Encoding: gzip
2024-11-19T08:55:01.0591830Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5
2024-11-19T08:55:01.0592076Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5
2024-11-19T08:55:01.0592599Z 2024-11-19T08:55:00.738Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: [DEBUG] GET https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview
2024-11-19T08:57:14.1037678Z 2024-11-19T08:57:14.103Z [DEBUG] provider.terraform-provider-azurerm_v4.10.0_x5: [ERR] GET https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview request failed: Get "https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview": dial tcp 4.150.240.10:443: connect: connection timed out
2024-11-19T08:57:14.1038912Z 2024-11-19T08:57:14.103Z [ERROR] provider.terraform-provider-azurerm_v4.10.0_x5: Response contains error diagnostic:
2024-11-19T08:57:14.1039261Z   diagnostic_summary=
2024-11-19T08:57:14.1039655Z   | retrieving Queue (Subscription: "XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX"
2024-11-19T08:57:14.1040020Z   | Resource Group Name: "RESOURCEGROUPNAME"
2024-11-19T08:57:14.1040334Z   | Namespace Name: "SERVICEBUS"
2024-11-19T08:57:14.1041260Z   | Queue Name: "QUEUENAME"): Get "https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview": dial tcp 4.150.240.10:443: connect: connection timed out
2024-11-19T08:57:14.1050062Z Queue Name: "QUEUENAME"): Get "https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview": dial tcp 4.150.240.10:443: connect: connection timed out
2024-11-19T08:57:14.1049106Z 2024-11-19T08:57:14.103Z [ERROR] vertex "module.servicebus_queue[\"id\"].azurerm_servicebus_queue.queue" error: retrieving Queue (Subscription: "XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX"
2024-11-19T08:57:14.1049362Z Resource Group Name: "RESOURCEGROUPNAME"
2024-11-19T08:57:14.1049540Z Namespace Name: "SERVICEBUS"
2024-11-19T08:57:14.1050062Z Queue Name: "QUEUENAME"): Get "https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview": dial tcp 4.150.240.10:443: connect: connection timed out

Expected Behaviour

After moving above v4.8.0 we experience timeouts on different resources. as example servicebus queue is used. We expect not to see any timeouts

Actual Behaviour

Get "https://management.azure.com/subscriptions/XXXXXXXXXX-XXXX-XXXX-XXXXXXXXXX/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ServiceBus/namespaces/SERVICEBUS/queues/QUEUENAME?api-version=2021-06-01-preview": dial tcp 4.150.240.10:443: connect: connection timed out

This is also for the other IP 4.150.241.10 which is also in the Debug log.

Steps to Reproduce

Agent is hosted on azure ubuntu vm 22.04 (docker agent) outbound traffic goes over the azure firewall

Run terraform plan.

Issue always occurs on large plans. For smaller plans this occurs only sometimes.

could this have something to do with the update of packages:

dependencies - update go-azure-sdk to v0.20241104.1140654 (https://github.com/hashicorp/terraform-provider-azurerm/issues/27896) dependencies - update go-azure-helpers to v0.71.0 (https://github.com/hashicorp/terraform-provider-azurerm/issues/27897) dependencies - update golang-jwt to v4.5.1 (https://github.com/hashicorp/terraform-provider-azurerm/issues/27938)

Important Factoids

No response

References

No response

DysonGates commented 1 day ago

We too are seeing the same issue. Screenshot 2024-11-21 110812

jackofallops commented 8 hours ago

Hi folks - this relates to a change that was made to the go-azure-sdk client library to be able to surface non-recoverable network failures. Whilst this went through significant testing, we're not able to cover every possible scenario. We're looking into this as a matter of urgency, so please bear with us.

Sanghamitra-PERSONAL commented 6 hours ago

@jackofallops :

We too are experiencing :

hashicorp / terraform-provider-azurerm