search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.6k stars 4.64k forks source link

Issues when deploying a Container App Environment with Internal LB and BYO Infrastructure Subnet #24583

Open JesperPVS opened 9 months ago

JesperPVS commented 9 months ago

Is there an existing issue for this?

Community Note

Terraform Version

1.7.0

AzureRM Provider Version

3.87.0

Affected Resource(s)/Data Source(s)

azurerm_container_app_environment

Terraform Configuration Files

variable "environment" {
  type        = string
  description = "Either dev, tst, pre, or pro"
}

variable "resource_group_base_name" {
  type    = string
  default = "rg-"
}

variable "location" {
  type    = string
  default = "West Europe"
}

data "azurerm_subnet" "container_app_env_subnet" {
  name                  = "ContainerApp-${var.environment}"
  virtual_network_name  = "vnet-prod"
  resource_group_name   = "network-pro"
}

data "azurerm_resource_group" "rg" {
  name = "${var.resource_group_base_name}${var.environment}"
}

resource "azurerm_log_analytics_workspace" "log_workspace" {
  name                = "${var.environment}-logs"
  location            = var.location
  resource_group_name = data.azurerm_resource_group.rg.name
  sku                 = "PerGB2018"
  retention_in_days   = 30

  lifecycle {
    ignore_changes = [
      tags,
    ]
  }
}

resource "azurerm_container_app_environment" "container_env" {
  name                            = "container-env-${var.environment}"
  resource_group_name             = data.azurerm_resource_group.rg.name
  location                        = var.location
  infrastructure_subnet_id        = data.azurerm_subnet.container_app_env_subnet.id
  internal_load_balancer_enabled  = true
  log_analytics_workspace_id      = azurerm_log_analytics_workspace.log_workspace.id

  workload_profile {
    name                  = "ca${var.environment}profile"
    workload_profile_type = "D4"
    minimum_count         = 1
    maximum_count         = 2
  }

  lifecycle {
    ignore_changes = [
      tags,
    ]
  }

  depends_on = [ 
    azurerm_log_analytics_workspace.log_workspace
  ]
}

Debug Output/Panic Output

2024-01-19T14:43:21.5821577Z azurerm_container_app_environment.container_env: Still creating... [8m41s elapsed]
2024-01-19T14:43:24.8542389Z 2024-01-19T14:43:24.853Z [DEBUG] provider.terraform-provider-azurerm_v3.88.0_x5.exe: AzureRM Request: 
2024-01-19T14:43:24.8550299Z GET /subscriptions/<Redacted>/providers/Microsoft.App/locations/westeurope/managedEnvironmentOperationStatuses/<Redacted>?api-version=2023-05-01&azureAsyncOperation=true&t=<Redacted> HTTP/1.1
2024-01-19T14:43:24.8557908Z Host: management.azure.com
2024-01-19T14:43:24.8559200Z User-Agent: Go/go1.21.3 (amd64-windows) go-autorest/v14.2.1 hashicorp/go-azure-sdk/managedenvironments/2023-05-01 HashiCorp Terraform/1.7.0 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.88.0 VSTS_d010764f-b06f-461c-a8ea-46749c10e8f1_Release__5023_26987_1 pid-<Redacted>
2024-01-19T14:43:24.8560367Z X-Ms-Correlation-Request-Id: 6b308699-0459-1f9c-b5e7-cb4d9ec7f307
2024-01-19T14:43:24.8561173Z Accept-Encoding: gzip: timestamp=2024-01-19T14:43:24.852Z
2024-01-19T14:43:25.4494889Z 2024-01-19T14:43:25.447Z [DEBUG] provider.terraform-provider-azurerm_v3.88.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/<Redacted>/providers/Microsoft.App/locations/westeurope/managedEnvironmentOperationStatuses/<Redacted>?api-version=2023-05-01&azureAsyncOperation=true&t=<Redacted>: 
2024-01-19T14:43:25.4506100Z HTTP/2.0 200 OK
2024-01-19T14:43:25.4507271Z Api-Supported-Versions: 2022-03-01, 2022-06-01-preview, 2022-10-01, 2022-11-01-preview, 2023-04-01-preview, 2023-05-01, 2023-05-02-preview, 2023-08-01-preview, 2023-11-02-preview
2024-01-19T14:43:25.4508023Z Cache-Control: no-cache
2024-01-19T14:43:25.4508478Z Content-Type: application/json; charset=utf-8
2024-01-19T14:43:25.4508968Z Date: Fri, 19 Jan 2024 14:43:25 GMT
2024-01-19T14:43:25.4509361Z Expires: -1
2024-01-19T14:43:25.4509717Z Pragma: no-cache
2024-01-19T14:43:25.4510242Z Server: Microsoft-IIS/10.0
2024-01-19T14:43:25.4513309Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2024-01-19T14:43:25.4514154Z Vary: Accept-Encoding,Accept-Encoding
2024-01-19T14:43:25.4514627Z X-Content-Type-Options: nosniff
2024-01-19T14:43:25.4515531Z X-Ms-Correlation-Request-Id: 6b308699-0459-1f9c-b5e7-cb4d9ec7f307
2024-01-19T14:43:25.4561268Z X-Ms-Ratelimit-Remaining-Subscription-Reads: 11960
2024-01-19T14:43:25.4562139Z X-Ms-Request-Id: b35eea93-b24b-4098-bcc2-7c19ab968dfd
2024-01-19T14:43:25.4562745Z X-Ms-Routing-Request-Id: SWEDENSOUTH:20240119T144325Z:b35eea93-b24b-4098-bcc2-7c19ab968dfd
2024-01-19T14:43:25.4563236Z X-Powered-By: ASP.NET
2024-01-19T14:43:25.4563432Z 
2024-01-19T14:43:25.4564641Z {"id":"/subscriptions/<Redacted>/providers/Microsoft.App/locations/westeurope/managedEnvironmentOperationStatuses/<Redacted>","name":"<Redacted>","status":"Failed","error":{"code":"OperationFailed","message":"Operation failed with general exception."},"startTime":"2024-01-19T14:34:45.1192071"}: timestamp=2024-01-19T14:43:25.445Z
2024-01-19T14:43:25.4591194Z 2024-01-19T14:43:25.448Z [ERROR] provider.terraform-provider-azurerm_v3.88.0_x5.exe: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58
2024-01-19T14:43:25.4592303Z   diagnostic_detail=
2024-01-19T14:43:25.4592782Z   | creating Managed Environment (Subscription: "<Redacted>"
2024-01-19T14:43:25.4593267Z   | Resource Group Name: "rg-pre"
2024-01-19T14:43:25.4593873Z   | Managed Environment Name: "container-env-pre"): polling after CreateOrUpdate: Code="OperationFailed" Message="Operation failed with general exception."
2024-01-19T14:43:25.4594978Z    diagnostic_severity=ERROR
2024-01-19T14:43:25.4595348Z   diagnostic_summary=
2024-01-19T14:43:25.4595806Z   | creating Managed Environment (Subscription: "<Redacted>"
2024-01-19T14:43:25.4596271Z   | Resource Group Name: "rg-pre"
2024-01-19T14:43:25.4596875Z   | Managed Environment Name: "container-env-pre"): polling after CreateOrUpdate: Code="OperationFailed" Message="Operation failed with general exception."
2024-01-19T14:43:25.4598779Z    tf_proto_version=5.4 tf_rpc=ApplyResourceChange tf_req_id=806f4305-7631-c7f6-24b1-07bd8556a4d4 @module=sdk.proto tf_provider_addr=provider tf_resource_type=azurerm_container_app_environment timestamp=2024-01-19T14:43:25.445Z
2024-01-19T14:43:25.4599989Z 2024-01-19T14:43:25.451Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-01-19T14:43:25.4601131Z 2024-01-19T14:43:25.451Z [ERROR] vertex "azurerm_container_app_environment.container_env" error: creating Managed Environment (Subscription: "<Redacted>"
2024-01-19T14:43:25.4601815Z Resource Group Name: "rg-pre"
2024-01-19T14:43:25.4602424Z Managed Environment Name: "container-env-pre"): polling after CreateOrUpdate: Code="OperationFailed" Message="Operation failed with general exception."
2024-01-19T14:43:25.4603176Z 2024-01-19T14:43:25.451Z [DEBUG] states/remote: state read serial is: 28; serial is: 28
2024-01-19T14:43:25.4604232Z 2024-01-19T14:43:25.451Z [DEBUG] states/remote: state read lineage is: 2ae87b0a-ef58-adb0-5ea1-585432e0eddc; lineage is: 2ae87b0a-ef58-adb0-5ea1-585432e0eddc
2024-01-19T14:43:25.4714933Z ╷
2024-01-19T14:43:25.4715780Z │ Error: creating Managed Environment (Subscription: "<Redacted>"
2024-01-19T14:43:25.4716375Z │ Resource Group Name: "rg-pre"
2024-01-19T14:43:25.4717097Z │ Managed Environment Name: "container-env-pre"): polling after CreateOrUpdate: Code="OperationFailed" Message="Operation failed with general exception."
2024-01-19T14:43:25.4717667Z │ 
2024-01-19T14:43:25.4718107Z │   with azurerm_container_app_environment.container_env,
2024-01-19T14:43:25.4718717Z │   on main.tf line 332, in resource "azurerm_container_app_environment" "container_env":
2024-01-19T14:43:25.4719504Z │  332: resource "azurerm_container_app_environment" "container_env" {
2024-01-19T14:43:25.4720598Z │ 
2024-01-19T14:43:25.4730812Z │ creating Managed Environment (Subscription:
2024-01-19T14:43:25.4732336Z │ "<Redacted>"
2024-01-19T14:43:25.4733975Z │ Resource Group Name: "rg-pre"
2024-01-19T14:43:25.4735674Z │ Managed Environment Name: "container-env-pre"): polling after
2024-01-19T14:43:25.4736917Z │ CreateOrUpdate: Code="OperationFailed" Message="Operation failed with
2024-01-19T14:43:25.4738142Z │ general exception."
2024-01-19T14:43:25.4739897Z ╵

Expected Behaviour

A Container App Environment with Internal Load Balancer and BYO Infrastructure subnet should be created, and ready for further tasks and/or configurations.

Actual Behaviour

Azure Container App Environment creation actually succeeds, as the resource is created in Azure, but Terraform fails on its polling, and thus thinks the resource failed to create.

The failure seems to happen when at a point, where I, in the Azure Portal, can see the resource is assigned an IP address from the infrastucture subnet. - I do not know where this causes the issue, just that the timing on the two things, seems to align.

Steps to Reproduce

terraform init

terraform plan

terraform apply

Important Factoids

No response

References

No response

tombuildsstuff commented 9 months ago

hey @JesperPVS

Thanks for opening this issue.

Taking a quick look into this, the error message that Terraform is surfacing here is coming from the Azure API:

Code="OperationFailed" Message="Operation failed with general exception."

.. and as such this issue would need to be fixed within the Container App Environment API, rather than something we can necessarily workaround within Terraform - out of interest does the Activity Log for this Resource show any additional information?

Thanks!

JesperPVS commented 9 months ago

hey @JesperPVS

Thanks for opening this issue.

Taking a quick look into this, the error message that Terraform is surfacing here is coming from the Azure API:

Code="OperationFailed" Message="Operation failed with general exception."

.. and as such this issue would need to be fixed within the Container App Environment API, rather than something we can necessarily workaround within Terraform - out of interest does the Activity Log for this Resource show any additional information?

Thanks!

Hey @tombuildsstuff,

In the Activity Log for the resource, the only thing it states is the following: image image For transparency; I deployed the resource again, to view the Activity Log, as I had deleted the one created from the output shown in the originating post.

The timestamp of the changed property is set between two statements in Terraform, as shown here: image (Don't mind the changed resource names, all settings are as stated in my initial post.)

Discrepancies in timestamps are due to timezones being GMT on our pipeline agents, while being GMT+1 on my local system.

JesperPVS commented 9 months ago

Hey @tombuildsstuff,

Found out what might possibly be the issue.. The subnet attribute for container app env in terraform seems to be case-sensitive, while the subnets in azure aren't.