azurerm_iothub_endpoint_eventhub StatusCode=504 -- Original Error: Code="GatewayTimeout"

[sergeibelov113]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.4.6

AzureRM Provider Version

3.58.0

Affected Resource(s)/Data Source(s)

azurerm_iothub_endpoint_eventhub

Terraform Configuration Files

resource "azurerm_iothub_endpoint_eventhub" "example" {
  for_each            = local.result_map
  resource_group_name = var.resource_group_name
  iothub_id           = azurerm_iothub.main[each.value["iothub_name"]].id
  name                = each.value["iothub_endpoint_eventhub"]["name"]
  authentication_type = "identityBased"
  entity_path         = lookup(each.value["iothub_endpoint_eventhub"], "entity_path", null)
  endpoint_uri        = lookup(each.value["iothub_endpoint_eventhub"], "endpoint_uri", null)
}

Debug Output/Panic Output

I have created a module which creates chain of DPS+multiple IoT Hubs.
This module also creates an EventHub for each IoT Hub in Message Routing/Route/Endpoints, but looks like something is wrong either with Azure Portal or with azurerm provider. Terraform constantly receives timeout after 11min 30 sec. All other resources like DPS and connected hubs are created successfully.

I have also checked that in the Azure portal I am able to add EventHub endpoint to IoT Hub without any issues manually. The system identity from IoT Hub is applied to the whole EventHub Namespace with a Contributor role. So there should not be any role issues. But Terraform still refuses to create this endppint in IoT Hubs. What have I missed?

Expected Behaviour

EventHub endpoints created

Actual Behaviour

EventHub endpoints not created created with the following error:

`Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:

• create

Terraform will perform the following actions:

module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["1"] will be created

• resource "azurerm_iothub_endpoint_eventhub" "example" {

  • authentication_type = "identityBased"
  • endpoint_uri = "sb://evh-id-converter-XXXXXXXXX.servicebus.windows.net"
  • entity_path = "evh-XXXXXXX"
  • id = (known after apply)
  • iothub_id = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/example-resources/providers/Microsoft.Devices/IotHubs/XXXXXXXXXXXXXX"
  • name = "XXXXXXXXXXXXXXXx"
  • resource_group_name = "example-resources" }

  module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["2"] will be created

• resource "azurerm_iothub_endpoint_eventhub" "example" {

  • authentication_type = "identityBased"
  • endpoint_uri = "sb://evh-id-converter-XXXXXXXXXXXXXX.servicebus.windows.net"
  • entity_path = "evh-id-converter-XXXXXXXXXX"
  • id = (known after apply)
  • iothub_id = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/example-resources/providers/Microsoft.Devices/IotHubs/XXXXXXXXXXXXXXXXXX"
  • name = "XXXXXXXXXXXXXXXXXX"
  • resource_group_name = "example-resources" }

  module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"] will be created

• resource "azurerm_iothub_endpoint_eventhub" "example" {
  • authentication_type = "identityBased"
  • endpoint_uri = "sb://evh-id-converter-XXXXXXXXXXXX.servicebus.windows.net"
  • entity_path = "evh-id-converter-XXXXXXXXXXx"
  • id = (known after apply)
  • iothub_id = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/example-resources/providers/Microsoft.Devices/IotHubs/XXXXXXXXXXXXXXXX"
  • name = "XXXXXXXXXXXXXXXXXX"
  • resource_group_name = "example-resources" }

Plan: 3 to add, 0 to change, 0 to destroy.`

module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"]: Still creating... [10m50s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"]: Still creating... [11m0s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["2"]: Still creating... [11m0s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["1"]: Still creating... [11m0s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["1"]: Still creating... [11m10s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"]: Still creating... [11m10s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["2"]: Still creating... [11m10s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["2"]: Still creating... [11m20s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["1"]: Still creating... [11m20s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"]: Still creating... [11m20s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["2"]: Still creating... [11m30s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"]: Still creating... [11m30s elapsed] module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["1"]: Still creating... [11m30s elapsed] ╷ │ Error: creating/updating Endpoint Eventhub: (Endpoint Name "id-sergei-converter" / Iot Hub Name "XXXXXXXXXXXXXXXXX" / Resource Group "example-resources"): devices.IotHubResourceClient#CreateOrUpdate: Failure sending request: StatusCode=504 -- Original Error: Code="GatewayTimeout" Message="The gateway did not receive a response from 'Microsoft.Devices' within the specified time period." │ │ with module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["3"], │ on iothub-dps-chain/main.tf line 83, in resource "azurerm_iothub_endpoint_eventhub" "example": │ 83: resource "azurerm_iothub_endpoint_eventhub" "example" { │ ╵ ╷ │ Error: creating/updating Endpoint Eventhub: (Endpoint Name "id-sergei-converter" / Iot Hub Name "XXXXXXXXXXXXXXXX" / Resource Group "example-resources"): devices.IotHubResourceClient#CreateOrUpdate: Failure sending request: StatusCode=504 -- Original Error: Code="GatewayTimeout" Message="The gateway did not receive a response from 'Microsoft.Devices' within the specified time period." │ │ with module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["2"], │ on iothub-dps-chain/main.tf line 83, in resource "azurerm_iothub_endpoint_eventhub" "example": │ 83: resource "azurerm_iothub_endpoint_eventhub" "example" { │ ╵ ╷ │ Error: creating/updating Endpoint Eventhub: (Endpoint Name "id-sergei-converter" / Iot Hub Name "XXXXXXXXXXXX" / Resource Group "example-resources"): devices.IotHubResourceClient#CreateOrUpdate: Failure sending request: StatusCode=504 -- Original Error: Code="GatewayTimeout" Message="The gateway did not receive a response from 'Microsoft.Devices' within the specified time period." │ │ with module.iothub-dps-chain.azurerm_iothub_endpoint_eventhub.example["1"], │ on iothub-dps-chain/main.tf line 83, in resource "azurerm_iothub_endpoint_eventhub" "example": │ 83: resource "azurerm_iothub_endpoint_eventhub" "example" { │

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[myc2h6o]

hi @sergeibelov113 thanks for opening the issue! From the error message there seems to be some issue within the Azure. For further troubleshooting, I'd suggest:

1. set TF_LOG to DEBUG, with this, the API request details including request URI and body as well as the correlation id will be printed out, which would be helpful with investigation.
2. when adding the endpoint on Azure Portal, you may open the developer window to capture the API requests.
3. with comparing the requests from Terraform and from Azure Portal, you may be able to find the difference which could be the root cause.

[sergeibelov113]

@myc2h6o Thanks I have compared two requests from portal and terraform, and some blocks are missed from terraform. ![] )

terraform plan shows however: ![] )

[myc2h6o]

@sergeibelov113 thanks for sharing the further details. Unfortunately, I'm not able to find anything new with it, but during my test, I'm able to repro the same error code and message by creating azurerm_iothub_endpoint_eventhub with a non-existing event hub. Would you mind double check if that's the same case? And below is a working example within my test environment, maybe you could give it a try in your environment and see if there is anything different. Lastly, the 504 error seems hiding some actual error at the service end, I'd suggest open a support ticket to Azure to see if they could find the root cause.

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "test"
  location = "westeurope"
}

resource "azurerm_eventhub_namespace" "test" {
  name                = "testnamespace"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  sku                 = "Basic"
}

resource "azurerm_eventhub" "test" {
  name                = "test"
  namespace_name      = azurerm_eventhub_namespace.test.name
  resource_group_name = azurerm_resource_group.test.name
  partition_count     = 2
  message_retention   = 1
}

resource "azurerm_iothub" "test" {
  name                = "test"
  resource_group_name = azurerm_resource_group.test.name
  location            = azurerm_resource_group.test.location

  sku {
    name     = "B1"
    capacity = "1"
  }

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_role_assignment" "test_azure_event_hubs_data_sender_system" {
  role_definition_name = "Azure Event Hubs Data Sender"
  scope                = azurerm_eventhub.test.id
  principal_id         = azurerm_iothub.test.identity[0].principal_id
}

resource "azurerm_iothub_endpoint_eventhub" "test" {
  resource_group_name = azurerm_resource_group.test.name
  iothub_id           = azurerm_iothub.test.id
  name                = "testendpoint"

  authentication_type = "identityBased"
  endpoint_uri        = "sb://${azurerm_eventhub_namespace.test.name}.servicebus.windows.net"
  entity_path         = azurerm_eventhub.test.name

  depends_on = [
    azurerm_role_assignment.test_azure_event_hubs_data_sender_system,
  ]
}