Storage Acccount Creation failure - Service Endpoint not available in Subnet

ssrahul96 commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.9

AzureRM Provider Version

3.3.0

Affected Resource(s)/Data Source(s)

azurerm_storage_account

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {}
  }
}

provider "azurerm" {
  features {}
}

data "azurerm_resource_group" "rg" {
  name = "my-poc-rg"
}

data "azurerm_virtual_network" "vnet" {
  name                = "my-VNET"
  resource_group_name = data.azurerm_resource_group.rg.name
}

data "azurerm_subnet" "vm-subnet" {
  name                 = "my-Subnet"
  resource_group_name  = data.azurerm_resource_group.rg.name
  virtual_network_name = data.azurerm_virtual_network.vnet.name
}

resource "azurerm_storage_account" "stg" {
  name                     = "stgissue1"
  resource_group_name      = data.azurerm_resource_group.rg.name
  location                 = data.azurerm_resource_group.rg.location
  account_kind             = "BlockBlobStorage"
  account_tier             = "Premium"
  account_replication_type = "LRS"
  min_tls_version          = "TLS1_2"

  network_rules {
    default_action             = "Deny"
    virtual_network_subnet_ids = [data.azurerm_subnet.vm-subnet.id]
  }
}

Debug Output/Panic Output

azurerm_storage_account.stg: Still creating... [10s elapsed]
azurerm_storage_account.stg: Still creating... [20s elapsed]
2022-04-26T13:30:56.592+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: AzureRM Request:
GET /subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/providers/Microsoft.Storage/locations/eastus/asyncoperations/4ca4f381-0f93-4b0c-a69e-5a8ec147f49c?monitor=true&api-version=2021-04-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.18.1 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v63.0.0 storage/2021-04-01 HashiCorp Terraform/1.1.9 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: b166e97f-7701-74d4-869c-c9f1eb7bec43
Accept-Encoding: gzip: timestamp=2022-04-26T13:30:56.592+0530
2022-04-26T13:30:56.858+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/providers/Microsoft.Storage/locations/eastus/asyncoperations/4ca4f381-0f93-4b0c-a69e-5a8ec147f49c?monitor=true&api-version=2021-04-01:
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 26 Apr 2022 08:00:55 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: b166e97f-7701-74d4-869c-c9f1eb7bec43
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11998
X-Ms-Request-Id: f9180172-069c-4af3-bda7-c8c848b7a0eb
X-Ms-Routing-Request-Id: SOUTHINDIA:20220426T080056Z:03e36e7b-18f0-4573-ac4d-2c0c298fb31e

{"status":"Failed","error":{"code":"NetworkAclsValidationFailure","message":"Validation of network acls failure: SubnetsHaveNoServiceEndpointsConfigured:Subnets cnatest-subnet of virtual network /subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/resourceGroups/devopsnow-poc-rg/providers/Microsoft.Network/virtualNetworks/CnaDev3-VNET do not have ServiceEndpoints for Microsoft.Storage resources configured. Add Microsoft.Storage to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Storage resources to these subnets.."}}: timestamp=2022-04-26T13:30:56.858+0530
2022-04-26T13:30:56.859+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: AzureRM Request:
GET /subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/resourceGroups/devopsnow-poc-rg/providers/Microsoft.Storage/storageAccounts/stgissue1?api-version=2021-04-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.18.1 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v63.0.0 storage/2021-04-01 HashiCorp Terraform/1.1.9 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: b166e97f-7701-74d4-869c-c9f1eb7bec43
Accept-Encoding: gzip: timestamp=2022-04-26T13:30:56.858+0530
2022-04-26T13:30:57.811+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/resourceGroups/devopsnow-poc-rg/providers/Microsoft.Storage/storageAccounts/stgissue1?api-version=2021-04-01:
HTTP/2.0 404 Not Found
Content-Length: 100
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 26 Apr 2022 08:00:56 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: b166e97f-7701-74d4-869c-c9f1eb7bec43
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11997
X-Ms-Request-Id: b7b9bf96-fb51-4870-9186-c2e25c1c7bff
X-Ms-Routing-Request-Id: SOUTHINDIA:20220426T080057Z:23638b49-9629-4bcd-a8ea-5e9796a2e3f6

{"error":{"code":"StorageAccountNotFound","message":"The storage account stgissue1 was not found."}}: timestamp=2022-04-26T13:30:57.811+0530
2022-04-26T13:30:57.812+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: Unlocking "azurerm_storage_account.stgissue1": timestamp=2022-04-26T13:30:57.811+0530
2022-04-26T13:30:57.812+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: Unlocked "azurerm_storage_account.stgissue1": timestamp=2022-04-26T13:30:57.811+0530
2022-04-26T13:30:57.825+0530 [ERROR] vertex "azurerm_storage_account.stg" error: retrieving Storage Account: (Name "stgissue1" / Resource Group "devopsnow-poc-rg"): storage.AccountsClient#GetProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="StorageAccountNotFound" Message="The storage account stgissue1 was not found."
╷
│ Error: retrieving Storage Account: (Name "stgissue1" / Resource Group "devopsnow-poc-rg"): storage.AccountsClient#GetProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="StorageAccountNotFound" Message="The storage account stgissue1 was not found."
│
│   with azurerm_storage_account.stg,
│   on main.tf line 26, in resource "azurerm_storage_account" "stg":
│   26: resource "azurerm_storage_account" "stg" {
│
╵
2022-04-26T13:30:57.830+0530 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-04-26T13:30:57.862+0530 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.3.0/windows_amd64/terraform-provider-azurerm_v3.3.0_x5.exe pid=73708
2022-04-26T13:30:57.862+0530 [DEBUG] provider: plugin exited

Expected Behaviour

The Storage account should be created

Actual Behaviour

The Storage account creation exits with issue

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

ssrahul96 commented 2 years ago

From the debug logs, i could see the below issue

{"status":"Failed","error":{"code":"NetworkAclsValidationFailure","message":"Validation of network acls failure: SubnetsHaveNoServiceEndpointsConfigured:Subnets cnatest-subnet of virtual network /subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/resourceGroups/devopsnow-poc-rg/providers/Microsoft.Network/virtualNetworks/CnaDev3-VNET do not have ServiceEndpoints for Microsoft.Storage resources configured. Add Microsoft.Storage to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Storage resources to these subnets.."}}

The Subnet i am associating does not have the storage service endpoint enabled.

Can we add ignore_missing_virtual_network_service_endpoint config under network_rules? Similar to EventHub?https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/eventhub_namespace#ignore_missing_virtual_network_service_endpoint

magodo commented 2 years ago

@ssrahul96 It appears you didn't specify the service_endpoints = ["Microsoft.Storage"] in your subnet config.

ssrahul96 commented 2 years ago

@ssrahul96 It appears you didn't specify the service_endpoints = ["Microsoft.Storage"] in your subnet config.

yea i ve noticed that and i ve fixed it at my end! i am raising this issue to throw that error message (or) provide an option to validate it

magodo commented 2 years ago

@ssrahul96

Unfortunately, the API definition of the network rule set for storage account doesn't has the same option ignore_missing_virtual_network_service_endpoint as the azurerm_eventhub_namespace: https://github.com/Azure/azure-rest-api-specs/blob/62631144da413d71bf332bd0104bbcbffe55d642/specification/storage/resource-manager/Microsoft.Storage/stable/2021-04-01/storage.json#L2376-L2383.

You are right, the error message is misleading. This is derived from the incorrect status code returned from the creation request of the storage account when the subnet is not correctly configured, where it returns 200, but it shouldn't:

2022-04-26T13:30:56.592+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: AzureRM Request:
GET /subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/providers/Microsoft.Storage/locations/eastus/asyncoperations/4ca4f381-0f93-4b0c-a69e-5a8ec147f49c?monitor=true&api-version=2021-04-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.18.1 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v63.0.0 storage/2021-04-01 HashiCorp Terraform/1.1.9 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
X-Ms-Correlation-Request-Id: b166e97f-7701-74d4-869c-c9f1eb7bec43
Accept-Encoding: gzip: timestamp=2022-04-26T13:30:56.592+0530
2022-04-26T13:30:56.858+0530 [DEBUG] provider.terraform-provider-azurerm_v3.3.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/providers/Microsoft.Storage/locations/eastus/asyncoperations/4ca4f381-0f93-4b0c-a69e-5a8ec147f49c?monitor=true&api-version=2021-04-01:
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 26 Apr 2022 08:00:55 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: b166e97f-7701-74d4-869c-c9f1eb7bec43
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11998
X-Ms-Request-Id: f9180172-069c-4af3-bda7-c8c848b7a0eb
X-Ms-Routing-Request-Id: SOUTHINDIA:20220426T080056Z:03e36e7b-18f0-4573-ac4d-2c0c298fb31e

{"status":"Failed","error":{"code":"NetworkAclsValidationFailure","message":"Validation of network acls failure: SubnetsHaveNoServiceEndpointsConfigured:Subnets cnatest-subnet of virtual network /subscriptions/6000915e-6d07-4eb2-9b61-5e03f46fb8fb/resourceGroups/devopsnow-poc-rg/providers/Microsoft.Network/virtualNetworks/CnaDev3-VNET do not have ServiceEndpoints for Microsoft.Storage resources configured. Add Microsoft.Storage to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Storage resources to these subnets.."}}: timestamp=2022-04-26T13:30:56.858+0530

ssrahul96 commented 2 years ago

Exacly, When i am running without DEBUG enabled, I am getting the below, error which is misleading,

azurerm_storage_account.stg: Creating...
azurerm_storage_account.stg: Still creating... [10s elapsed]
azurerm_storage_account.stg: Still creating... [20s elapsed]
╷
│ Error: retrieving Storage Account: (Name "stgissue1" / Resource Group "my-poc-rg"): storage.AccountsClient#GetProperties: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="StorageAccountNotFound" Message="The storage account stgissue1 was not found."
│
│   with azurerm_storage_account.stg,
│   on main.tf line 26, in resource "azurerm_storage_account" "stg":
│   26: resource "azurerm_storage_account" "stg" {
│
╵

Is this because of incorrect state code? that is returned from the previous API?

magodo commented 2 years ago

@ssrahul96 Yes, the error you see above is a later point after creating the storage account. But the provider should have been errored out during creation.

ssrahul96 commented 2 years ago

@ssrahul96 Yes, the error you see above is a later point after creating the storage account. But the provider should have been errored out during creation.

Should a issue be created in https://github.com/Azure/azure-rest-api-specs ?

magodo commented 2 years ago

@ssrahul96 Exactly, I'm working on that! Also trying to provide a workarond in the provider 👍

ssrahul96 commented 2 years ago

@ssrahul96 Exactly, I'm working on that! Also trying to provide a workarond in the provider 👍

Sure Thanks

magodo commented 2 years ago

The upstream issue is filed here: https://github.com/Azure/azure-rest-api-specs/issues/18844

dc232 commented 1 year ago

Did the work around work for anyone by having service_endpoints = ["Microsoft.Storage"] specified?

ssrahul96 commented 1 year ago

@dc232 yes, it should work after adding the service_endpoint

dc232 commented 1 year ago

Thanks @ssrahul96

Auxority commented 10 months ago

Ran into this issue today, thanks @dc232 for your comment. That fixed it for me :) The error message could've been a little bit clearer though.

seddon commented 8 months ago

Ran into this issue today, thanks @dc232 for your comment. That fixed it for me :) The error message could've been a little bit clearer though.

Same here, thanks @dc232

GavBurke commented 6 months ago

Is there anyone else running into this issue not had @dc232's workaround work for them?!?

The org I work for we have an infrastructure team who deploys the vnets, subnets etc., and that has the Microsoft.Storage enabled for those, and have been enabled for a good couple of years now, but yet I'm trying to recreate one of our deployments (destroyed the old SA and redeployed due to changes in the replication type, which can't be amended when using premium) and now getting this same error, but the vnet/subnet are all configured correctly...

hashicorp / terraform-provider-azurerm