search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.51k stars 4.6k forks source link

Cannot create azurerm_storage_container in azurerm_storage_account that uses network_rules #2977

Open phil-bevan opened 5 years ago

phil-bevan commented 5 years ago

Community Note

Terraform (and AzureRM Provider) Version

Terraform v0.11.11

Affected Resource(s)

Terraform Configuration Files

resource "azurerm_storage_account" "test-storage-acct" {
  name                     = "${var.prefix}storacct"
  resource_group_name      = "${var.resgroup}"
  location                 = "${var.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  network_rules {
    ip_rules                   = ["aaa.bbb.ccc.ddd/ee"]
    virtual_network_subnet_ids = ["${var.subnetid}"]
  }
}
resource "azurerm_storage_container" "provisioning" {
  name                  = "${var.prefix}-provisioning"
  resource_group_name   = "${var.resgroup}"
  storage_account_name  = "${azurerm_storage_account.test-storage-acct.name}"
  container_access_type = "private"
}

Debug Output

dkuzmenok commented 1 year ago

@tombuildsstuff @magodo This issue is here for 3.5 years and I cannot see the end of it. It is blocking huge amount of work for storage accounts that are forced to use a firewall.

Is it possible to make any "temporary" solution until MSFT will implement something from their side? Bringing something to terraform provider takes one week (until next release is there), and waiting for MSFT already takes much more :)

RobertFloor commented 1 year ago

As others have said, Terraform uses private urls for management of the file share. In our cases DNS resolving of these endpoints was not working correctly. You can test this by lookingup the url using terraform console and then investigate the resource. Use nslookupto determine if you can resolve the url. if not there are several options. For example you could add them to your /etc/hosts file. This solved it our case. Another option is to add the private link to a private dns zon and forward you dns to this private zone.

tspearconquest commented 1 year ago

https://learn.microsoft.com/en-us/azure/dns/dns-private-resolver-overview can also help if you're using private endpoints. You may need this to do the forwarding mentioned by @RobertFloor

DerTim1 commented 1 year ago

If someone need to work around this issue for a storage account of type "FileStorage", e.g. for a NFS-Share, this example code worked for us (based on the previous replies with deployment templates):

resource "azurerm_storage_account" "example-nfs" {
  name                      = "examplenfs"
  resource_group_name       = azurerm_resource_group.example.name
  location                  = azurerm_resource_group.example.location
  account_tier              = "Premium"
  account_kind              = "FileStorage"
  account_replication_type  = "LRS"
  enable_https_traffic_only = false

  network_rules {
    default_action             = "Deny"
    # ip_rules                   = ["127.0.0.1/24"]
    virtual_network_subnet_ids = [azurerm_subnet.example_subnet_1]
    bypass                     = ["AzureServices"]
  }
}

# NOTE Normally, we will do the following azurerm_storage_share.
#  Due to https://github.com/hashicorp/terraform-provider-azurerm/issues/2977
#  this isn't possible right now. So we working around with an ARM template, see
#  post https://github.com/hashicorp/terraform-provider-azurerm/issues/2977#issuecomment-875693407
# resource "azurerm_storage_share" "example-nfs_fileshare" {
#   name                 = "example"
#   storage_account_name = azurerm_storage_account.example-nfs.name
#   quota                = 100
#   enabled_protocol     = "NFS"
# }
resource "azurerm_resource_group_template_deployment" "example-nfs_fileshare" {
  name                = "${azurerm_storage_account.example-nfs.name}-fileshare-example"
  resource_group_name = azurerm_resource_group.example.name
  deployment_mode     = "Incremental"

  parameters_content = jsonencode({
    "storage_account_name" = {
      value = azurerm_storage_account.example-nfs.name
    }
    "fileshare_name" = {
      value ="example"
    }
  })
  template_content = <<TEMPLATE
{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storage_account_name": {
      "type": "string"
    },
    "fileshare_name": {
      "type": "string"
    }
  },
  "variables": {},
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts/fileServices/shares",
      "name": "[concat(parameters('storage_account_name'), '/default/', parameters('fileshare_name'))]",
      "apiVersion": "2021-02-01",
      "properties": {
        "shareQuota": 100,
        "enabledProtocols": "NFS"
      }
    }
  ]
}
TEMPLATE

  depends_on = [azurerm_storage_account.example-nfs]
}
abindg commented 1 year ago

Hi I think I am also facing the same issue . In my case , I had to import a NFS enabled storage account with ip filters . When I am running terraform plan after the import from my local development environment it is working fine . But when I am running plan from Azure pipelines I am getting the error .

retrieving Container "**" (Account "" / Resource Group ""): containers.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailure" Message="This request is not authorized to perform this operation.\n"**

I have added the agent's public IP to the firewall rules of the storage account , but it still does not work .

Please let me know , if anyone has found a solution to make this working from the pipeline

bstalte-Avanade commented 1 year ago

This issue still exists in the latest provider (3.51.0). You cannot manage container blob files if you set the storage account network_rules.default_action = Deny. I've tried setting the public IP to my runners public IP as well as the Vnet/Subnet integration with no luck. The provider must be attempting to view the blobs via public endpoint only.

(https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account#default_action)

resource "azurerm_storage_account" "example" { name = "storageaccountname" resource_group_name = azurerm_resource_group.example.name

location = azurerm_resource_group.example.location account_tier = "Standard" account_replication_type = "LRS"

network_rules { default_action = "Deny" ip_rules = ["100.0.0.1"] virtual_network_subnet_ids = [azurerm_subnet.example.id] }

abindg commented 1 year ago

But I just wanted to understand , why this same config is successfully applied from local development , but fails on the pipeline , when in both cases there is a public ip attached and both the public IPs have been whitelisted .

This doesnot seem to be a terraform problem , In that case it wouldnot have worked locally

robertbrandso commented 1 year ago

@abindg Do you use self-hosted Azure DevOps Agents? If yes, where are the agents located? On-premises or Azure? If in Azure, are they hosted in the same Azure region as the storage account?

egorshulga commented 1 year ago

Eventually we found the reason why. So, the thing is that Storage Accounts have a feature, that all requests to them are routed not via Internet, but via internal Microsoft networks. This only happens when a requesting agent and the Storage Account are in the same region.

In our case we are using Microsoft-hosted agents in our ci. The CI agents are given to us in the same region as the region of Azure DevOps organisation. That is why it happened that both the agents and our Storage Accounts live in the same region.

It seems to be a "feature" from Microsoft, when they want to route requests for Storage Accounts in the fastest way, because it is assumed that there could be a heavy load.

magodo commented 1 year ago

@abindg IIRC, when you are running in the pipeline, the agent will not use its public ip for the outgoing requests due to some reason that I'm not so sure.

abindg commented 1 year ago

@robertbrandso the agents on hosted on azure . I will check the region . Probably it differs .

So if I understand correctly , then for Microsoft internal based routing from agent to storage account they need to be in the same region.

Is this understanding correct

magodo commented 1 year ago

@abindg Probably the reverse way, that if they are in the same region, then it will use some internla/private routing instead (I might be lying).

tspearconquest commented 1 year ago

When they are in the same region, then the traffic is NOT routed via internet and when they are in the same region, then it is. The Storage Account Network Rules applies only to traffic routed via the internet, so for same-region requests, the fix is to use a private endpoint. You need a private endpoint in the storage account. See this comment for more details.

You can also refer here for more about connecting devops to a storage account private endpoint

abindg commented 1 year ago

When they are in the same region, then the traffic is NOT routed via internet and when they are in the same region, then it is. The Storage Account Network Rules applies only to traffic routed via the internet, so for same-region requests, the fix is to use a private endpoint. You need a private endpoint in the storage account. See this comment for more details.

You can also refer here for more about connecting devops to a storage account private endpoint

Thanks . I will check this .

magodo commented 1 year ago

@tspearconquest, @abindg Check out this: https://github.com/hashicorp/terraform-provider-azurerm/issues/12411#issuecomment-877452093. I didn't mean the request goes through the private Azure backbone network (as PE), but is still through the public internet, just the source ip address is not the public ip of the agent, but a private ip.

mvervoort commented 1 year ago

That's also our observation. When fetching the public IP address of the Azure hosted agent (by issuing a request to https://ifconfig.me/ip) and whitelisting this on the Storage Account, it only has effect when both are in a different region. Could there be any way to fetch the IP address of the Azure hosted agent, which is used 'internally'?

robertbrandso commented 1 year ago

@mvervoort, it wouldn't have any effect if you did, because you can't add RFC1918 adresses to your storage account firewall.

bstalte-Avanade commented 1 year ago

Thanks guys. I didn't realize my runner being used was in the same region as the storage account, hence the problem. Once I assigned a different region runner to process the job and added that public IP to the new storage account ip rules, it was able to provision the account and list/add the container blobs.

Easiest solution is to have runner/agents in multiple regions, allowing the code to reference another regions runner if you're locking down the storage account being managed by the Terraform deployment.

abindg commented 1 year ago

thanks everyone . I got my pipeline agents to read the containers now .

As my agent and the storage account were in the same region , so I added the subnet of my pipeline hosted agents to the network rule of the storage account , first manually and then brought it under my terraform state by including it in my terraform manifest .

It works fine now .

Thanks again.

randompixel commented 1 year ago

For those of us using Terraform Cloud to manage the runs, this issue is quite infuriating. Seeing I don't understand the private network stuff well enough (some official Terraform documentation on this would be amazing), the workaround I'm using is to use Azure CLI to turn off the network rules just before the TF runs in the Pipeline then turn it back on again afterwards. Not ideal however as TF is no longer fully managing state.

I doubt even adding the current agent to the firewall rules WITHIN the Terraform provider would work as it would try and get state before doing anything and fail.

scott1138 commented 1 year ago

Check out this article https://developer.hashicorp.com/terraform/tutorials/cloud/cloud-agents

Sent from my Commodore 64

From: David Harper @.> Sent: Friday, April 21, 2023 9:14:32 AM To: hashicorp/terraform-provider-azurerm @.> Cc: scott1138 @.>; Manual @.> Subject: Re: [hashicorp/terraform-provider-azurerm] Cannot create azurerm_storage_container in azurerm_storage_account that uses network_rules (#2977)

For those of us using Terraform Cloud to manage the runs, this issue is quite infuriating. Seeing I don't understand the private network stuff well enough (some official Terraform documentation on this would be amazing), the workaround I'm using is to use Azure CLI to turn off the network rules just before the TF runs in the Pipeline then turn it back on again afterwards. Not ideal however as TF is no longer fully managing state.

I doubt even adding the current agent to the firewall rules WITHIN the Terraform provider would work as it would try and get state before doing anything and fail.

— Reply to this email directly, view it on GitHubhttps://github.com/hashicorp/terraform-provider-azurerm/issues/2977#issuecomment-1517901029, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHHSHHAGHRGZ35AE3OB5UADXCKI4RANCNFSM4G24DG2A. You are receiving this because you are subscribed to this thread.Message ID: @.***>

vinisdl commented 1 year ago

I'm having this problem/bug

jsoliveir commented 1 year ago

For those of us using Terraform Cloud to manage the runs, this issue is quite infuriating. Seeing I don't understand the private network stuff well enough (some official Terraform documentation on this would be amazing), the workaround I'm using is to use Azure CLI to turn off the network rules just before the TF runs in the Pipeline then turn it back on again afterwards. Not ideal however as TF is no longer fully managing state.

I doubt even adding the current agent to the firewall rules WITHIN the Terraform provider would work as it would try and get state before doing anything and fail.

Yes, this is what I am, more or less, doing too. 👍 It's not ideal but it's a good workaround.

I also have fully private infrastructure. In my pipeline I use the azure cli before running terraform in order to apply a network rule that allows the runner IP address to talk to the storage account.

After terraform finishes its job I run the azcli again to drop the network rule created before.

syedhasanraza commented 1 year ago

Another easy solution is to fetch the ip address of the agent in pipeline and add it to the storage account firewall. This will allow us to add containers etc to storage account, since the agent ip will be whitelisted.

`

storage account defined in .tf file

    resource "azurerm_storage_account" "storage" {
    name                          = var.storageaccount
    resource_group_name           = var.resourcegroup
    location                      = var.environment_location
    account_tier                  = "Standard"
    account_replication_type      = "LRS"
    public_network_access_enabled = true
    enable_https_traffic_only     = true

    network_rules {
      default_action = "Deny"
      virtual_network_subnet_ids  = [data.azurerm_subnet.appservice_subnet.id]
      ip_rules                    = [var.ip_address]  # add the variable to ip rules for whitelisting
    }
  }

'

.yml file stage

 - stage: DeployDEV
        displayName: 'Deploy to DEV'
        condition: succeeded()
        dependsOn: InstallTerraform
        variables:
        - group: 'Dev-Grp'
        - name: TF_VAR_ip_address  # here we have to prefix the variable 'ip_address' with TF_VAR_ 
          value: '$(ip_address)'
        jobs:
          - job: PlanAndApply
            displayName: Init, Plan and Apply
            continueOnError: false
            steps:

    # fetch the agent pipeline ip address and set the value of variable ip_address defined in .tfvars file
        - task: AzureCLI@2
          displayName: Set the IP Adress Variable
          inputs:
            azureSubscription: '$(serviceconnection)'
            scriptType: pscore
            scriptLocation: inlineScript
            addSpnToEnvironment: true
            inlineScript: |
              $ip = (((invoke-webrequest -Uri 'http://checkip.amazonaws.com/').Content | % {[char]$_}) -join "").TrimEnd()
              Write-Host "##vso[task.setvariable variable=ip_address]$ip"

    # replace token task will replace the value of all variables in .tfvar file using variable group.(No need to add ip_address in variable group, but it needs to be defined in .tfvar file
    # i.e. ip_address ="$(ip_address)" )
        - task: qetza.replacetokens.replacetokens-task.replacetokens@5
          displayName: Replace values
          inputs:
            targetFiles: |
              *.tfvars
            encoding: 'auto'
            tokenPattern: 'azpipelines'
            writeBOM: true
            actionOnMissing: 'warn'
            keepToken: false
            actionOnNoFiles: 'fail'
            enableTransforms: false
            enableRecursion: false
            useLegacyPattern: false
            enableTelemetry: true

        - task: TerraformTaskV4@4
          displayName: Init Terraform
          inputs:
            provider: 'azurerm'
            command: 'init'
            backendServiceArm : '$(serviceconnection)'
            backendAzureRmResourceGroupName: '$(tfstate_resourcegroup)'
            backendAzureRmStorageAccountName: '$(tfstate_storageaccount)' 
            backendAzureRmContainerName: '$(tfstate_container)'
            backendAzureRmKey: '$(tfstate_key)'

        - task: TerraformTaskV4@4
          displayName: Plan
          inputs:
            provider: 'azurerm'
            command: 'plan'
            backendServiceArm: '$(serviceconnection)' 
            environmentServiceNameAzureRM : '$(serviceconnection)'
            backendAzureRmResourceGroupName: '$(tfstate_resourcegroup)'
            backendAzureRmStorageAccountName: '$(tfstate_storageaccount)' 
            backendAzureRmContainerName: '$(tfstate_container)'
            backendAzureRmKey: '$(tfstate_key)'
            commandOptions: '-var-file="terraform.tfvars" -out=plan.out'
            publishPlanResults: "plan.out"

        - task: TerraformCLI@0
          displayName: Apply
          inputs:
            command: 'apply'
            backendServiceArm: '$(serviceconnection)'
            backendAzureRmResourceGroupName: '$(tfstate_resourcegroup)'
            backendAzureRmStorageAccountName: '$(tfstate_storageaccount)'
            allowTelemetryCollection: true

`

egorshulga commented 1 year ago

@syedhasanraza that means, that there will always be one IP in the whitelisting list, right? Even after provisioning, it will still be there till the next provisioning

syedhasanraza commented 1 year ago

Yes, that's correct,but you can remove it after the storage is provisioned.

egorshulga commented 1 year ago

@syedhasanraza no, we can't, as it will then break refresh of the containers 😕

Actually, this limitation forced us to use AzAPI provider for containers. We use Azure DevOps and MS-hosted agents, so we can whitelist an agent IP, but only for provisioning (only in the same job). All whitelisting "holes" are removed immediately in the same job or stage.

bacatta commented 1 year ago

Since it's, by far, the most voted issue and a very old one. I think you should close this as "will not be fixed" AND add a warning note on azurerm_storage_share resource about this "know upstream issue", with a link to this bug.

494206 commented 1 year ago

If I understand this and the other related threads, successfully managing private endpoints on storage accounts with Terraform is currently only possible:

Seconding what @bacatta said -- there really should be a warning on the documentation page for azurerm_storage_account, about this.

I'd go a little further and have that note state that private endpoints are currently not 'officially' supported for storage accounts by the Azurerm provider. It would be a stretch to argue otherwise at the moment, IMHO.

magodo commented 1 year ago

@494206 I think you confused private endpoint vs network rule (firewall). The original 403 error is from network rule. For how to setup the private endpoint for the storage account and its sub resources, you can reference this.

494206 commented 1 year ago

@magodo I understand how private endpoints work, but your criticism is fair - the issue isn't just the private endpoints.

The storage account firewall has a default "allow" rule when each storage account is deployed. This allows Internet connection attempts to any of the built-in public service endpoints on storage accounts. (~6 endpoints per storage account)

If I understand this and the other threads, "azurerm" will fail if a "Deny" rule is set, unless there's a private endpoint it can reach for each of the services per storage account. (~6 private endpoints per storage account?)

It's time to leave a note in the documentation -- 4 years this issue has been open.

(edit: if there's another workaround that I've missed, would be happy to hear it.)

magodo commented 1 year ago

@494206 I didn't meant to criticise about your comment, just ensure things are clear :)

There is a workaround to use the azapi provider to manage the storage container using pure its management plane API, which won't be impacted by any DNS resolve/firewall issues: https://github.com/hashicorp/terraform-provider-azurerm/issues/2977#issuecomment-1127384422

petr-stupka commented 1 year ago

Just reading the issue from top to the bottom about storage account configuration and i'm not sure if TF is the right place for this challenge (i would not call it issue) as TF have no possibility to change the way how Azure services works.

It all goes down to how Azure Storage Account works and it's services like blobs and files and how it works when private / service endpoint is enabled on Storage Account. I agree the MS Docs topics about Storage Account services and configuration are not easy to understand, however i think it is a must to have this knowledge when enabling private endpoint / service endpoint.

Private Endpoint Service Endpoint Limitations

So basically there are some options available:

I just want to share my experience and this is how we are doing that and we are pretty happy with those options so far.

Happy coding!

494206 commented 11 months ago

There is a workaround to use the azapi provider to manage the storage container using pure its management plane API, which won't be impacted by any DNS resolve/firewall issues: #2977 (comment)

This should be added as a note on the azurerm_storage_account page, as the workaround is "don't use azurerm".

mikew3432 commented 11 months ago

Like to add a note from the field, as I have encountered this kind of issue a lot myself in the last week. This is a wider issue I would describe as:

Is there anywhere azurerm providers document their dependency on the data plane API instead of ARM management plane API - as sometimes it's not obvious (e.g. azurerm_storage_data_lake_gen2_filesystem)

ajaykrishnanchn commented 11 months ago

This issue was fixed for me with the latest AzureRm provider version I used version 3.70.0

awood86 commented 11 months ago

This issue was fixed for me with the latest AzureRm provider version I used version 3.70.0

Was it? I'm trying to create a container after my storage account has been created with a private link and it won't allow me. I'm on 3.73.0 too

ajaykrishnanchn commented 11 months ago

This issue was fixed for me with the latest AzureRm provider version I used version 3.70.0

Was it? I'm trying to create a container after my storage account has been created with a private link and it won't allow me. I'm on 3.73.0 too

If you are running from the pipeline make sure you have added pipeline IP in network restriction, Also add 'Storage Blob Data Contributor ' to your current objected (data.azurerm_client_config.current.object_id)

awood86 commented 11 months ago

This issue was fixed for me with the latest AzureRm provider version I used version 3.70.0

Was it? I'm trying to create a container after my storage account has been created with a private link and it won't allow me. I'm on 3.73.0 too

If you are running from the pipeline make sure you have added pipeline IP in network restriction, Also add 'Storage Blob Data Contributor ' to your current objected (data.azurerm_client_config.current.object_id)

I understand that and totally appreciate that option would work. It's probably that I'm coming from a different angle in that if you deploy via Bicep you don't need to open up any IPs as this is all done via the API.

Obviously Bicep and Terraform are two different products with different ways of working, so I'll just have to adjust accordingly

rijulg commented 10 months ago

I ran into this problem today as well, and did not want to go via the template route. As a result I wrote the following using the azapi provider

# main.tf
resource "random_uuid" "acl" {}

# Can't use the terraform azurerm provider because that uses the storage api directly
# and since the storage account may (most probably) has public access disabled that API fails
# Documentation for this is available at
# https://learn.microsoft.com/en-us/azure/templates/microsoft.storage/2022-09-01/storageaccounts/fileservices/shares?pivots=deployment-language-terraform
resource "azapi_resource" "this" {
  type      = "Microsoft.Storage/storageAccounts/fileServices/shares@2022-09-01"
  name      = var.name
  parent_id = "${var.storage_account_id}/fileServices/default"
  body = jsonencode({
    properties = {
      accessTier       = var.access_tier
      enabledProtocols = var.enabled_protocol
      shareQuota       = var.claims_storage_quota_gb
      signedIdentifiers = [
        {
          accessPolicy = {
            permission = var.permissions
          }
          id = random_uuid.acl.result
        }
      ]
    }
  })
  response_export_values = ["*"]
}

# vars.tf
variable "name" { type = string }
variable "storage_account_id" { type = string }
variable "access_tier" { type = string }
variable "enabled_protocol" { type = string }
variable "claims_storage_quota_gb" { type = number }
variable "permissions" { type = string }

this works just fine with the service principal

Joerg-L commented 10 months ago

We are facing the same issue and are really surprised that there is until now no good solution existing for that kind of issues.

We have the Azure Policy "Storage Accounts should disable public network access" enabled with "deny" and so it's even not possible atm to allow public/restricted network access to the storage account for deploying the containers.

brianandrus commented 10 months ago

I ran into this awhile back and would manually create the containers. I finally figured out the simplified AzAPI code and am posting it here. This will create a storage account that disables public access and enables NFSv3, then create a container in that account.

resource "azurerm_storage_account" "group_blob_storage" {
  name                      = "example_storage_account"
  resource_group_name       = local.app_rg_name
  location                  = local.location
  account_kind              = "StorageV2"
  account_tier              = "Standard"
  access_tier               = "Hot"
  account_replication_type  = "LRS"
  enable_https_traffic_only = true
  is_hns_enabled            = true
  nfsv3_enabled             = true
  min_tls_version           = "TLS1_2"
  allow_blob_public_access  = false
  tags                      = local.default_tags
  lifecycle {
    ignore_changes = [
      tags["CreationDate"],
    ]
  }
  network_rules {
    default_action = "Deny"
  }
}

resource "azapi_resource" "group_blob_containers" {
  type      = "Microsoft.Storage/storageAccounts/blobServices/containers@2022-09-01"
  name      = "mycontainer"
  parent_id = "${azurerm_storage_account.group_blob_storage.id}/blobServices/default"
  body = jsonencode({
    properties = {
      defaultEncryptionScope      = "$account-encryption-key"
      denyEncryptionScopeOverride = false
      enableNfsV3AllSquash        = false
      enableNfsV3RootSquash       = false
      metadata                    = {}
      publicAccess                = "None"
    }
  })
  depends_on = [
    azurerm_storage_account.group_blob_storage
  ]
}

You can change the json-encoded properties as needed.

cailyoung commented 9 months ago

I can confirm that it very much looks like azapi solves the similar problem we have; i.e. how to have Terraform add containers to storage accounts that do not have public internet ingress for the data plane.

Is it possible for this provider to rework the resource to use the resource management API for this operation? The RM API is internet-accessible, which means we don't have to do anything with network rules regarding where terraform apply is executing.

kewalaka commented 9 months ago

There is another reason for using AZAPI that is related to this issue. It surfaces when you disable storage account Shared Key authentication, as per "well architected framework" guidance:

In order to make this work with Terraform, you need to add storage_use_azuread = true to your provider block, i.e. something like this:

provider "azurerm" {
  features {}
  storage_use_azuread        = true
}

This changes the behaviour of Terraform so that instead of fetching the shared keys and using those, it uses EntraID/AzureAD permissions of the principal running Terraform.

Then, if you try and create a container using azurerm_storage_container it will fail, but if you use the AZAPI provider, it works. Similar reasons to the firewall problem noted in this issue.

Obviously, if you want to do any data plane operations, you would need to set IAM appropriately, however if you just want to create containers and set permissions on them, AZAPI works fine.

The GitHub action in this repository illustrates the behaviour of AZAPI vs AzureRM for this use case.

It's worth noting there are some limitations when disabling shared key authentication, e.g. when using the table & files API as per the AzureRM provider documentation, however this approach works well for things like Terraform state (or any blob workload), and is useful where alignment to WAF principles is a requirement.

guderkar commented 7 months ago

Creating container using azurerm provider works just fine when public access to storage is disabled and private endpoint is setup correctly. This provider uses data plane operations to do so (i.e. blob.core.windows.net / dfs.core.windows.net API calls).

Most of the corporate setups are

So what conserns me more than creating container is creating the azurerm_storage_account resource which does those data plane operations on its creation. This means when I have this NO_PROXY variable set the deployment agent will resolve the data plane API call to public IP (because private endpoint does not exist yet) and goes directly via internal network which is not ment for internet access -> will get timeout and fail. This is kind of a vicious cycle because private endpoint cannot be created before the storage account.

There are two cures for this problem.

Btw exact same problem are having other reources as well, e.g. key vault.

fabio-s-franco commented 5 months ago

This has been a nightmare, due to static website properties it tries to read. If public network access is disabled, you're done. Good luck.

It will only work if you get the setup right during creation, if there is anything wrong, or a DNS resolution is a mismatch it is not possible to fix without some manual trickery.

It is also not possible to have it created with dynamic IPs as this requires that the private endpoint gets created before the A record and that creates the chicken and egg scenario.

Whilst the issue with data plane is not resolved, can't a flag be set somewhere, anywhere, to forcefully discard anything related with static website properties? If I don't need it, won't use it, I would like to be allowed to disregard it. Since this would be an opt-in option, it wouldn't introduce any breaking change and would bring peace to this very annoying issue.

I'd say that to make a change like this not be an ugly temporary patch, you could introduce a concept that applies to all resources of the provider, such as something like "global_options", that could be an open text field for situations like this.

Edit: It just came to me, I could simply use:

lifecycle {
    ignore_changes  = [static_website]
  }

🤯

Except, it doesn't help. Still tries to retrieve static website properties. But it could be a solution... @tombuildsstuff I saw your comment here (https://github.com/hashicorp/terraform-provider-azurerm/issues/20257#issuecomment-1461352568), shouldn't something like the above do exactly what it was being asked, albeit explicitly? I am surprised I haven't thought of that myself before and also that this doesn't seem to matter for the provider. If ignore_changes is analogous to not track you mentioned, this would make it very explicit, for whoever wants to do so and actually fits terraform's resource model.

michasacuer commented 4 months ago

Local development indeed is a nightmare.

Scenario:

And we have an error:

Error: checking for existing Container "test-container" (Account "Account \"accountname\" (IsEdgeZone false / ZoneName \"\" / Subdomain Type \"blob\" / DomainSuffix \"core.windows.net\")"): executing request: unexpected status 403 (403 This request is not authorized to perform this operation.) with AuthorizationFailure: This request is not authorized to perform this operation.
│ RequestId:e0f6fe03-801e-0015-5776-9c1a69000000
│ Time:2024-05-02T09:50:15.9741260Z
│
│   with azurerm_storage_container.container[0],
│   on main.tf line 49, in resource "azurerm_storage_container" "container":
│   49: resource "azurerm_storage_container" "container" {

Any idea when it can be fixed? like @guderkar said, that setup is kinda common in the industry

guderkar commented 3 months ago

I once tested with Azure VPN Gateway + Client and I remember that there are two things essential

https://learn.microsoft.com/en-us/azure/vpn-gateway/azure-vpn-client-optional-configurations

jkroepke commented 2 months ago

Whilst we may look to split the Data Plane functionality out to a separate resource in the future, unfortunately doing so breaks some scenarios (for example, guaranteeing that there's no static website configured, which would need to be done within the main azurerm_storage_account resource).

Originally posted by @tombuildsstuff in https://github.com/hashicorp/terraform-provider-azurerm/issues/26542#issuecomment-2209269652

In near future, we may see an 4.0 which should allow to break things, right?

guaranteeing that there's no static website configured

It's sometime that could be unmanaged azurerm_storage_account and having a dedicated resource like azurerm_storage_account_static_website_configuration?

Similar to https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_website_configuration

gburkecw commented 1 month ago

I had a similar issue with creating file shares in a storage account that has public access disabled. The Azure Portal allowed me to create the file share through the public internet, so I knew it was possible to do so through the Azure API. The public access restriction only applies to accessing the data in the share, not creating the share.

Anyways, to workaround this I created the storage account with the AzureRM provider as normal, with public access disabled. Then I use the AzApi provider to circumvent the bug/limitation in the AzureRM provider and hit the AzApi directly to manage the resource:

resource "azapi_resource" "myshare" {
  type      = "Microsoft.Storage/storageAccounts/fileServices/shares@2023-05-01"
  name      = "myshare"
  parent_id = "${azurerm_storage_account.st-smbshare.id}/fileServices/default"

  body = {
    properties = {
      accessTier : "Hot",
      shareQuota : 1024,
      enabledProtocols : "SMB"
    }
  }
}

Took a little bit of tweaking and looking at the exported template, but the key was to get the parent_id right. I imagine for a blob container this would be blobServices instead of fileServices, but the concept should be similar.

Now for blob services, it might not actually be possible to create the container over the public internet with public access disabled, you'll have to confirm that in the portal. But since it was working in the portal for file shares, I knew that it SHOULD work through TF one way or the other.

Note that using this method, state is managed properly. I tested by deleting the share after it was created and TF plan/apply recreated it properly. There are other areas where I've been using the AzAPI provider as an "escape hatch" when something is broken or unsupported in AzureRM provider.