Fails to create azure container right after azure storage account was created

sekasara commented 2 years ago

Getting The specified resource does not exist error while creating azure container right after azure storage account was created

Source Code :

resource "azurerm_storage_account" "storage_account" {
  name                                = "mystgacnt45612"
  resource_group_name     = "tf-azure"
  location                            = "East US"
  account_tier                     = "Standard"
  account_replication_type = "LRS"
  allow_blob_public_access = true
}

resource "azurerm_storage_container" "storage_container" {
  storage_account_name  = azurerm_storage_account.storage_account.name
  name                              = "mystgcntr45612"
  container_access_type    = "blob"
}

Actual Behaviour :

While executing the above script, getting below error

│ Error: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:652e8c77-701e-008f-024c-2a1e0c000000\nTime:2022-02-25T13:31:39.3693903Z"
│
│   with azurerm_storage_container.storage_container,
│   on main.tf line 10, in resource "azurerm_storage_container" "storage_container":
│   10: resource "azurerm_storage_container" "storage_container" {
│

Expected Behaviour :

It should create the both storage account and storage container.

sekasara commented 2 years ago

any updates here ???

magodo commented 2 years ago

@sekasara Thank you for reporting this! Is this systematic or intermittent? I've tried with above config with no issue. If this is systematic, then would you please provide the debug log for us to anlyze? Otherwise, this might due to some Azure issue.

sekasara commented 2 years ago

@magodo i tried with other system too. but still getting same issue. getting issue while executing terraform apply --auto-approve

magodo commented 2 years ago

@sekasara Would you please follow the debugging guide and setup the environment variables before you run terraform apply, then paste a link to a gist with the generated log? BTW, which region do you run Terraform from?

zeyangli commented 2 years ago

I had the same problem, I created an AzureCloud resource using terraform in China @magodo

Macbook Pro M1 Pro

azurerm_storage_container.tfstate: Creating...
╷
│ Error: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:583c3681-d01e-0030-7ab9-8c920f000000\nTime:2022-06-30T19:45:39.5361718Z"
│ 
│   with azurerm_storage_container.tfstate,
│   on main.tf line 19, in resource "azurerm_storage_container" "tfstate":
│   19: resource "azurerm_storage_container" "tfstate" {

Azure config info

➜  backend az config  get   
Command group 'config' is experimental and under development. Reference and support levels: https://aka.ms/CLI_refstatus
{
  "cloud": [
    {
      "name": "name",
      "source": "/Users/lizeyang/.azure/config",
      "value": "AzureCloud"
    }
  ],
  "core": [
    {
      "name": "first_run",
      "source": "/Users/lizeyang/.azure/config",
      "value": "yes"
    }
  ]
}

DEBUG log

2022-07-01T04:01:13.154+0800 [DEBUG] provider.terraform-provider-azurerm_v3.10.0_x5: AzureRM Response for https://management.azure.com/subscriptions/2ca5488f-5097-42cd-80a1-cecfd8ced33e/resourceGroups/tfdevopszeyang1/providers/Microsoft.Storage/storageAccounts/tfstateadmin111/listKeys?%24expand=kerb&api-version=2021-04-01: 
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 30 Jun 2022 20:01:12 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 14de9f02-f6f7-d025-238a-e8c0b65d0676
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 11998
X-Ms-Request-Id: 817dbf05-1f86-4f46-ad26-836abd6aaaa7
X-Ms-Routing-Request-Id: KOREACENTRAL:20220630T200113Z:d708b251-3486-4cc9-9a24-87e6b924e59d

{"keys":[{"creationTime":"2022-06-30T20:00:43.5959648Z","keyName":"key1","value":"s21XcHDI8plc5VNSFUkwoTOcwJq8Y9bLCwC4dQlXOiPuokbPQp5vOyAap+bAk9s7I2kYW5M8jYXf+AStwOft5g==","permissions":"FULL"},{"creationTime":"2022-06-30T20:00:43.5959648Z","keyName":"key2","value":"tNkOOFd4gAjWQsMV8dsr/l/4fPNBQOvAIQARuOcG6Y2kA3dOpLXUYPCo79HrpSCq8J/2J2+3BLdY+AStB+ib7A==","permissions":"FULL"}]}: timestamp=2022-07-01T04:01:13.154+0800
azurerm_storage_account.tfstate: Still creating... [40s elapsed]
2022-07-01T04:01:15.508+0800 [DEBUG] provider.terraform-provider-azurerm_v3.10.0_x5: Unlocking "azurerm_storage_account.tfstateadmin111": timestamp=2022-07-01T04:01:15.508+0800
2022-07-01T04:01:15.508+0800 [DEBUG] provider.terraform-provider-azurerm_v3.10.0_x5: Unlocked "azurerm_storage_account.tfstateadmin111": timestamp=2022-07-01T04:01:15.508+0800
2022-07-01T04:01:15.510+0800 [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an unexpected new value for azurerm_storage_account.tfstate, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .edge_zone: was null, but now cty.StringVal("")
azurerm_storage_account.tfstate: Creation complete after 41s [id=/subscriptions/2ca5488f-5097-42cd-80a1-cecfd8ced33e/resourceGroups/tfdevopszeyang1/providers/Microsoft.Storage/storageAccounts/tfstateadmin111]
azurerm_storage_container.tfstate: Creating...
2022-07-01T04:01:15.533+0800 [INFO]  Starting apply for azurerm_storage_container.tfstate
2022-07-01T04:01:15.534+0800 [DEBUG] azurerm_storage_container.tfstate: applying the planned Create change
2022-07-01T04:01:15.534+0800 [DEBUG] provider.terraform-provider-azurerm_v3.10.0_x5: setting computed for "metadata" from ComputedKeys: timestamp=2022-07-01T04:01:15.534+0800
2022-07-01T04:01:16.739+0800 [INFO]  provider.terraform-provider-azurerm_v3.10.0_x5: Creating Container "tfstate" in Storage Account "tfstateadmin111": timestamp=2022-07-01T04:01:16.738+0800
2022-07-01T04:01:17.988+0800 [ERROR] vertex "azurerm_storage_container.tfstate" error: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:066c5f0b-801e-002d-54bc-8ce68e000000\nTime:2022-06-30T20:01:17.9347383Z"
╷
│ Error: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:066c5f0b-801e-002d-54bc-8ce68e000000\nTime:2022-06-30T20:01:17.9347383Z"
│ 
│   with azurerm_storage_container.tfstate,
│   on main.tf line 19, in resource "azurerm_storage_container" "tfstate":
│   19: resource "azurerm_storage_container" "tfstate" {
│ 
╵
2022-07-01T04:01:18.009+0800 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-07-01T04:01:18.013+0800 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.10.0/darwin_arm64/terraform-provider-azurerm_v3.10.0_x5 pid=14510
2022-07-01T04:01:18.014+0800 [DEBUG] provider: plugin exited

magodo commented 2 years ago

I've tried 10 times with above config, and all worked.

@zeyangli What is an AzureCloud? Also want to confirm whether you are targeting the Azure public cloud or the mooncake (Azure China)? Regarding the log, would you please provide the full log? I'd like to see which API failed (only DEBUG level is needed). Besides, does this problem systematically happen?

zeyangli commented 2 years ago

@magodo

Hello, I am using Terraform configured with azure_cli. I'm not using the Chinese version.

full logs: https://github.com/zeyangli/devops-on-k8s/blob/master/tf.log

magodo commented 2 years ago

@zeyangli From the error log, the error occurs when creating the container via its data plane API:

Status=404 Code="ResourceNotFound" Message="The specified resource does not exist

Does this happens systematically when you recreate a new storage account and container, or only happens to this existing one? If it is the latter, then looks like a service issue, which you can ask Azure support to look into.

DanielSpindler83 commented 1 year ago

I too am having the same problem. This issue is systematic and repeatable. Debug log

I apply the below and it fails to create the container with the error shown below. I run apply again and it creates the container successfully. I run destroy. I then repeat with the same results each time. Tested 4 or so times.

terraform {
    required_version = "~> 1.3.3"
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "=3.28.0"
    }
  }
}

provider "azurerm" {
    subscription_id = "XXXXXXXXXX"
    client_id = "XXXXXXXX"
    client_secret = "XXXXXX"
    tenant_id = "XXXXX"
    features {}
}

resource "azurerm_resource_group" "app_grp" {
    name = "app-grp"
    location = "Australia East"
}

resource "azurerm_storage_account" "app_grp" {
  name                     = "appgrpdantest19831208"
  resource_group_name      = "app-grp"
  location                 = "Australia East"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"
  depends_on = [
    azurerm_resource_group.app_grp
  ]
}

resource "azurerm_storage_container" "data" {
  name                  = "data"
  storage_account_name  = "appgrpdantest19831208"
  container_access_type = "blob"
  depends_on = [
    azurerm_storage_account.app_grp
  ]
}

resource "azurerm_storage_blob" "maintf" {
  name                   = "main.tf"
  storage_account_name   = "appgrpdantest19831208"
  storage_container_name = "data"
  type                   = "Block"
  source                 = "main.tf"
  depends_on = [
    azurerm_storage_container.data
  ]
}

Error to stderr

azurerm_resource_group.app_grp: Creating...
azurerm_resource_group.app_grp: Creation complete after 2s [id=/subscriptions/cf8310cd-2b1f-4e63-9678-9e53daf74b50/resourceGroups/app-grp]
azurerm_storage_account.app_grp: Creating...
azurerm_storage_account.app_grp: Still creating... [10s elapsed]
azurerm_storage_account.app_grp: Still creating... [20s elapsed]
azurerm_storage_account.app_grp: Creation complete after 28s [id=/subscriptions/cf8310cd-2b1f-4e63-9678-9e53daf74b50/resourceGroups/app-grp/providers/Microsoft.Storage/storageAccounts/appgrpdantest19831208]
azurerm_storage_container.data: Creating...
╷
│ Error: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:9b8b3286-601e-0028-0f4a-ec3e8f000000\nTime:2022-10-30T10:29:22.9087723Z"       
│
│   with azurerm_storage_container.data,
│   on main.tf line 36, in resource "azurerm_storage_container" "data":
│   36: resource "azurerm_storage_container" "data" {
│
╵

DEBUG Log Error

2022-10-30T20:29:22.725+1000 [ERROR] provider.terraform-provider-azurerm_v3.28.0_x5.exe: Response contains error diagnostic: @module=sdk.proto diagnostic_detail= tf_proto_version=5.2 tf_provider_addr=provider tf_resource_type=azurerm_storage_container @caller=github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/diag/diagnostics.go:56 diagnostic_severity=ERROR diagnostic_summary="failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:9b8b3286-601e-0028-0f4a-ec3e8f000000\nTime:2022-10-30T10:29:22.9087723Z"" tf_req_id=b839d5ae-d0ae-b8f4-d22a-c43127ecac3a tf_rpc=ApplyResourceChange timestamp=2022-10-30T20:29:22.725+1000
2022-10-30T20:29:22.732+1000 [ERROR] vertex "azurerm_storage_container.data" error: failed creating container: failed creating container: containers.Client#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The specified resource does not exist.\nRequestId:9b8b3286-601e-0028-0f4a-ec3e8f000000\nTime:2022-10-30T10:29:22.9087723Z"

Apply again after the error - and the container is created successfully.

azurerm_storage_container.data: Creating...
azurerm_storage_container.data: Creation complete after 1s [id=https://appgrpdantest19831208.blob.core.windows.net/data]
azurerm_storage_blob.maintf: Creating...
azurerm_storage_blob.maintf: Creation complete after 0s [id=https://appgrpdantest19831208.blob.core.windows.net/data/main.tf]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Something I would like to highlight that appears many times in my logs, that was also present in @zeyangli 's logs is:

2022-10-30T20:29:22.493+1000 [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an unexpected new value for azurerm_storage_account.app_grp, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .edge_zone: was null, but now cty.StringVal("")

I am using Azure in Australia East location.

Let me know if you need any further info.

magodo commented 1 year ago

This might duplicates to #13070. See my reply: https://github.com/hashicorp/terraform-provider-azurerm/issues/13070#issuecomment-1296562402.

RSwarnkar commented 1 year ago

Same issue facing. This is so annoying.

Plan: 6 to add, 0 to change, 0 to destroy.

Error: shares.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailure" Message="This request is not authorized to perform this operation.\nRequestId:6afc2182-001a-005e-1a37-a8014e000000\nTime:2023-06-26T14:05:27.6158591Z"

  with module.storageaccount.azurerm_storage_share.sashare["sa0-fer-hot-bpah-fshare-ungravity01"],
  on ../modules/landingzone/BPAH/storageaccount/main.tf line 58, in resource "azurerm_storage_share" "sashare":
  58: resource "azurerm_storage_share" "sashare" {

Nail-Sv commented 11 months ago

Hi folks, any update to this thema?

sabrite commented 10 months ago

I have faced this number of times with provider version 3.69. The storage account is created but when creating the container, I get the following error:

Error: reading blob properties for AzureRM Storage Account "stgtestivpwebfiles": storage.BlobServicesClient#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ContainerOperationFailure" Message="The specified resource does not exist.\nRequestId:6ece5383-801e-009d-7586-df9fcc000000\nTime:2023-09-04T23:21:52.5714020Z"

molinto commented 8 months ago

Still an issue on v=3.79.0

Resource Group Name: "MAIN-RG-SHARRY"
│ Storage Account Name: "samainsharry"): storage.BlobServicesClient#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ContainerOperationFailure" Message="The specified resource does not exist.\nRequestId:3fa02587-701e-0081-3af1-1ded28000000\nTime:2023-11-23T09:41:36.8089880Z"
│ 
│   with azurerm_storage_account.main_storage,
│   on main.tf line 58, in resource "azurerm_storage_account" "main_storage":
│   58: resource "azurerm_storage_account" "main_storage" {

mpenndev commented 6 months ago

Can consistently recreate with this minimal example:

resource "azurerm_storage_account" "this" {
  name                     = "testaccount"
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "test" {
  name                 = test
  storage_account_name = azurerm_storage_account.this.name
}

Always fails with:

storage.BlobServicesClient#GetServiceProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ContainerOperationFailure" Message="The specified resource does not exist.

But commenting out the container for the 1st run and then introducing it on a 2nd run creates successfully.

mpenndev commented 6 months ago

As magodo referenced this does seem to be a DNS caching issue.

I've found this to be a reliable (created from scratch 5 times in a row) work-around:

resource "random_string" "id" {
  length  = 4
  upper   = false
  special = false
}

resource "azurerm_storage_account" "this" {
  name                     = "testaccount${random_string.id.result}"
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "test" {
  name                 = test
  storage_account_name = azurerm_storage_account.this.name
}

hashicorp / terraform-provider-azurerm