search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.59k stars 4.63k forks source link

Cannot delete a non empty file share storage #7834

Closed charlesdee closed 7 months ago

charlesdee commented 4 years ago

Community Note

Terraform (and AzureRM Provider) Version

Terraform v0.12.28

provider.azurerm v2.18.0
provider.null v2.1.2

Affected Resource(s)

azurerm_storage_share_directory azurerm_storage_share

Terraform Configuration Files

# #storage account and file share
resource "azurerm_storage_account" "test" {
  name                     = local.storage_name
  resource_group_name      = azurerm_resource_group.test.name
  location                 = azurerm_resource_group.test.location
  account_tier             = var.account_tier
  account_replication_type = var.account_replication_type
}

resource "azurerm_storage_share" "test" {
  name                 = azurerm_storage_account.test.name
  storage_account_name = azurerm_storage_account.test.name
  quota                = var.storage_fileshare_quota 
}

Debug Output

Error: Error deleting Storage Share "test" (File Share "testfilestracc" / Account "testfilestracc" / Resource Group "testfilestraccqa"): directories.Client#Delete: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status= Code="DirectoryNotEmpty" Message="The specified directory is not empty.\nRequestId:b6fa48eb-a01a-0055-4059-5f4f8e000000\nTime:2020-07-21T12:23:36.3720820Z"

Panic Output

Expected Behavior

should have deleted the file share

Actual Behavior

throws an error saying the directory is not empty, isn't possible to delete a non-empty directory? Any workaround or solution would be appreciated! i used terraform to create storage account , fileshare and the directories as well.

Steps to Reproduce

  1. terraform apply

Important Factoids

NO

References

lrxtom2 commented 4 years ago

@charlesdee Thanks for opening the issue! I am working on that to look for workaround about that. Thanks for that.

pragadeeshraju commented 4 years ago

@lrxtom2 this is happening when you create the directories for share using "azurerm_storage_share_directory" and destroying the created resources(when there is some files in the directory)

whindes commented 4 years ago

Any updates on this? I've tried using a null_reference local-exec that calls the Azure CLI: az storage file delete-batch with no luck. **EDIT *. This was actually an issue using azurerm_storage_share_directory and not necessarily the original issue post.

yupwei68 commented 4 years ago

Hi @charlesdee , thanks for opening this issue. Sorry that currently I can't reproduce this error. I have tested with a azurerm_storage_share with files and directories, and deleted them successfully with Terraform destroy. Would you mind retrying and if the error still runs out, please providing a more detailed reproducing steps.

pragadeeshraju commented 4 years ago

Hello @yupwei68 thanks for looking at this.

i tried again and bumped into the same issue

azurerm_storage_share_directory.share_directory["config"]: Destroying... [id=https://sharedeletest.file.core.windows.net/sharedeletest/config]

Error: Error deleting Storage Share "config" (File Share "sharedeletest" / Account "sharedeletest" / Resource Group "sharedeletest"): directories.Client#Delete: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="DirectoryNotEmpty" Message="The specified directory is not empty.\nRequestId:bbf8e95b-501a-0038-5314-a17210000000\nTime:2020-10-13T03:51:10.1935325Z"

cc @mybayern1974

pragadeeshraju commented 4 years ago

steps to replicate this:

1) Create storage account using terraform 2) create storage share 3) create storage share directory 4) Destroy it (terraform destory)


 resource "azurerm_storage_account" "storage_account" {
   name                     = local.storage_name
   resource_group_name      = azurerm_resource_group.rg.name
   location                 = azurerm_resource_group.rg.location
   account_tier             = var.account_tier
   account_replication_type = var.account_replication_type
 }

 resource "azurerm_storage_share" "storage_share" {
   name                 = azurerm_storage_account.storage_account.name
   storage_account_name = azurerm_storage_account.storage_account.name
   quota                = var.storage_fileshare_quota 
 }

 resource "azurerm_storage_share_directory" "share_directory" {
   for_each             = toset(var.share_directories)
   name                 = each.value
   share_name           = azurerm_storage_account.storage_account.name
   storage_account_name = azurerm_storage_share.storage_share.name
   #depends_on           = [azurerm_storage_share.storage_share]
 }
yupwei68 commented 4 years ago

Hi @pragadeeshraju , I've tried your configuration. And I destroy it successfully. Would you try the latest Azurerm verion v2.31.1 and see if the error still runs out?

...
Destroy complete! Resources: 4 destroyed.
pragadeeshraju commented 4 years ago

Hi @yupwei68 I used the 2.31.1 version and got the same error. please check the below steps to replicate the issue.

I think you might have missed adding some files to the directory and then destroy it( sorry I missed to mention it in my earlier comment)

1) create the following resources

 terraform apply -var-file=var.tfvars

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_resource_group.rg will be created
  + resource "azurerm_resource_group" "rg" {
      + id       = (known after apply)
      + location = "australiaeast"
      + name     = "sharedeletest"
    }

  # azurerm_storage_account.storage_account will be created
  + resource "azurerm_storage_account" "storage_account" {
      + access_tier                      = (known after apply)
      + account_kind                     = "StorageV2"
      + account_replication_type         = "LRS"
      + account_tier                     = "Standard"
      + allow_blob_public_access         = false
      + enable_https_traffic_only        = true
      + id                               = (known after apply)
      + is_hns_enabled                   = false
      + large_file_share_enabled         = (known after apply)
      + location                         = "australiaeast"
      + min_tls_version                  = "TLS1_0"
      + name                             = "sharedeletest"
      + primary_access_key               = (sensitive value)
      + primary_blob_connection_string   = (sensitive value)
      + primary_blob_endpoint            = (known after apply)
      + primary_blob_host                = (known after apply)
      + primary_connection_string        = (sensitive value)
      + primary_dfs_endpoint             = (known after apply)
      + primary_dfs_host                 = (known after apply)
      + primary_file_endpoint            = (known after apply)
      + primary_file_host                = (known after apply)
      + primary_location                 = (known after apply)
      + primary_queue_endpoint           = (known after apply)
      + primary_queue_host               = (known after apply)
      + primary_table_endpoint           = (known after apply)
      + primary_table_host               = (known after apply)
      + primary_web_endpoint             = (known after apply)
      + primary_web_host                 = (known after apply)
      + resource_group_name              = "sharedeletest"
      + secondary_access_key             = (sensitive value)
      + secondary_blob_connection_string = (sensitive value)
      + secondary_blob_endpoint          = (known after apply)
      + secondary_blob_host              = (known after apply)
      + secondary_connection_string      = (sensitive value)
      + secondary_dfs_endpoint           = (known after apply)
      + secondary_dfs_host               = (known after apply)
      + secondary_file_endpoint          = (known after apply)
      + secondary_file_host              = (known after apply)
      + secondary_location               = (known after apply)
      + secondary_queue_endpoint         = (known after apply)
      + secondary_queue_host             = (known after apply)
      + secondary_table_endpoint         = (known after apply)
      + secondary_table_host             = (known after apply)
      + secondary_web_endpoint           = (known after apply)
      + secondary_web_host               = (known after apply)

      + blob_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + delete_retention_policy {
              + days = (known after apply)
            }
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }

      + network_rules {
          + bypass                     = (known after apply)
          + default_action             = (known after apply)
          + ip_rules                   = (known after apply)
          + virtual_network_subnet_ids = (known after apply)
        }

      + queue_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + hour_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }

          + logging {
              + delete                = (known after apply)
              + read                  = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
              + write                 = (known after apply)
            }

          + minute_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }
        }
    }

  # azurerm_storage_share.storage_share will be created
  + resource "azurerm_storage_share" "storage_share" {
      + id                   = (known after apply)
      + name                 = "sharedeletest"
      + quota                = 50
      + resource_manager_id  = (known after apply)
      + storage_account_name = "sharedeletest"
      + url                  = (known after apply)
    }

  # azurerm_storage_share_directory.share_directory["config"] will be created
  + resource "azurerm_storage_share_directory" "share_directory" {
      + id                   = (known after apply)
      + name                 = "config"
      + share_name           = "sharedeletest"
      + storage_account_name = "sharedeletest"
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_resource_group.rg: Creating...
azurerm_resource_group.rg: Creation complete after 2s [id=/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest]
azurerm_storage_account.storage_account: Creating...
azurerm_storage_account.storage_account: Still creating... [10s elapsed]
azurerm_storage_account.storage_account: Still creating... [20s elapsed]
azurerm_storage_account.storage_account: Creation complete after 30s [id=/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest/providers/Microsoft.Storage/storageAccounts/sharedeletest]
azurerm_storage_share.storage_share: Creating...
azurerm_storage_share.storage_share: Creation complete after 1s [id=https://sharedeletest.file.core.windows.net/sharedeletest]
azurerm_storage_share_directory.share_directory["config"]: Creating...
azurerm_storage_share_directory.share_directory["config"]: Still creating... [10s elapsed]
azurerm_storage_share_directory.share_directory["config"]: Still creating... [20s elapsed]
azurerm_storage_share_directory.share_directory["config"]: Still creating... [30s elapsed]
azurerm_storage_share_directory.share_directory["config"]: Still creating... [40s elapsed]
azurerm_storage_share_directory.share_directory["config"]: Creation complete after 41s [id=https://sharedeletest.file.core.windows.net/sharedeletest/config]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

2) upoad some files to config folder (azurerm_storage_share_directory.share_directory["config"]: Creation complete after 41s)

3) try to destroy the same

$ terraform destroy -var-file=var.tfvars

azurerm_resource_group.rg: Refreshing state... [id=/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest]
azurerm_storage_account.storage_account: Refreshing state... [id=/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest/providers/Microsoft.Storage/storageAccounts/sharedeletest]
azurerm_storage_share.storage_share: Refreshing state... [id=https://sharedeletest.file.core.windows.net/sharedeletest]
azurerm_storage_share_directory.share_directory["config"]: Refreshing state... [id=https://sharedeletest.file.core.windows.net/sharedeletest/config]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # azurerm_resource_group.rg will be destroyed
  - resource "azurerm_resource_group" "rg" {
      - id       = "/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest" -> null
      - location = "australiaeast" -> null
      - name     = "sharedeletest" -> null
      - tags     = {} -> null
    }

  # azurerm_storage_account.storage_account will be destroyed
  - resource "azurerm_storage_account" "storage_account" {
      - access_tier                    = "Hot" -> null
      - account_kind                   = "StorageV2" -> null
      - account_replication_type       = "LRS" -> null
      - account_tier                   = "Standard" -> null
      - allow_blob_public_access       = false -> null
      - enable_https_traffic_only      = true -> null
      - id                             = "/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest/providers/Microsoft.Storage/storageAccounts/sharedeletest" -> null
      - is_hns_enabled                 = false -> null
      - location                       = "australiaeast" -> null
      - min_tls_version                = "TLS1_0" -> null
      - name                           = "sharedeletest" -> null
      - primary_access_key             = (sensitive value)
      - primary_blob_connection_string = (sensitive value)
      - primary_blob_endpoint          = "https://sharedeletest.blob.core.windows.net/" -> null
      - primary_blob_host              = "sharedeletest.blob.core.windows.net" -> null
      - primary_connection_string      = (sensitive value)
      - primary_dfs_endpoint           = "https://sharedeletest.dfs.core.windows.net/" -> null
      - primary_dfs_host               = "sharedeletest.dfs.core.windows.net" -> null
      - primary_file_endpoint          = "https://sharedeletest.file.core.windows.net/" -> null
      - primary_file_host              = "sharedeletest.file.core.windows.net" -> null
      - primary_location               = "australiaeast" -> null
      - primary_queue_endpoint         = "https://sharedeletest.queue.core.windows.net/" -> null
      - primary_queue_host             = "sharedeletest.queue.core.windows.net" -> null
      - primary_table_endpoint         = "https://sharedeletest.table.core.windows.net/" -> null
      - primary_table_host             = "sharedeletest.table.core.windows.net" -> null
      - primary_web_endpoint           = "https://sharedeletest.z8.web.core.windows.net/" -> null
      - primary_web_host               = "sharedeletest.z8.web.core.windows.net" -> null
      - resource_group_name            = "sharedeletest" -> null
      - secondary_access_key           = (sensitive value)
      - secondary_connection_string    = (sensitive value)
      - tags                           = {} -> null

      - network_rules {
          - bypass                     = [
              - "AzureServices",
            ] -> null
          - default_action             = "Allow" -> null
          - ip_rules                   = [] -> null
          - virtual_network_subnet_ids = [] -> null
        }

      - queue_properties {

          - hour_metrics {
              - enabled               = true -> null
              - include_apis          = true -> null
              - retention_policy_days = 7 -> null
              - version               = "1.0" -> null
            }

          - logging {
              - delete                = false -> null
              - read                  = false -> null
              - retention_policy_days = 0 -> null
              - version               = "1.0" -> null
              - write                 = false -> null
            }

          - minute_metrics {
              - enabled               = false -> null
              - include_apis          = false -> null
              - retention_policy_days = 0 -> null
              - version               = "1.0" -> null
            }
        }
    }

  # azurerm_storage_share.storage_share will be destroyed
  - resource "azurerm_storage_share" "storage_share" {
      - id                   = "https://sharedeletest.file.core.windows.net/sharedeletest" -> null
      - metadata             = {} -> null
      - name                 = "sharedeletest" -> null
      - quota                = 50 -> null
      - resource_manager_id  = "/subscriptions/sharedeletest-subcription/resourceGroups/sharedeletest/providers/Microsoft.Storage/storageAccounts/sharedeletest/fileServices/default/shares/sharedeletest" -> null
      - storage_account_name = "sharedeletest" -> null
      - url                  = "https://sharedeletest.file.core.windows.net/sharedeletest" -> null
    }

  # azurerm_storage_share_directory.share_directory["config"] will be destroyed
  - resource "azurerm_storage_share_directory" "share_directory" {
      - id                   = "https://sharedeletest.file.core.windows.net/sharedeletest/config" -> null
      - metadata             = {} -> null
      - name                 = "config" -> null
      - share_name           = "sharedeletest" -> null
      - storage_account_name = "sharedeletest" -> null
    }

Plan: 0 to add, 0 to change, 4 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

azurerm_storage_share_directory.share_directory["config"]: Destroying... [id=https://sharedeletest.file.core.windows.net/sharedeletest/config]

Error: Error deleting Storage Share "config" (File Share "sharedeletest" / Account "sharedeletest" / Resource Group "sharedeletest"): directories.Client#Delete: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="DirectoryNotEmpty" Message="The specified directory is not empty.\nRequestId:664d7e66-e01a-000f-5c36-a1ed91000000\nTime:2020-10-13T07:55:01.7833642Z"

Hope this helps

yupwei68 commented 3 years ago

Hi @pragadeeshraju ,the service team confirms that the storage share directory can not be deleted when it's not empty. So as the limitation on portal. Please refer to the service api definition: https://docs.microsoft.com/en-us/rest/api/storageservices/delete-directory

tombuildsstuff commented 3 years ago

@yupwei68 whilst the Azure Portal contains this limitation, since Terraform has requested confirmation there's no reason why Terraform couldn't avoid this limitation by listing & deleting any files within the Directory when we go to delete it

setagana commented 3 years ago

Agreed, it would be great if there was a allow_force_delete boolean argument on the azurerm_storage_share_directory resource.

My use case: I need to spin up a Container Instance that mounts a file share containing some seed data. The container, once running, will add and modify files in that file share. Once it does that, I'm no longer able to tear down my environment with terraform because of this issue.

razgrim commented 1 year ago

Hi @pragadeeshraju ,the service team confirms that the storage share directory can not be deleted when it's not empty. So as the limitation on portal. Please refer to the service api definition: https://docs.microsoft.com/en-us/rest/api/storageservices/delete-directory

That's a lazy answer

@yupwei68 whilst the Azure Portal contains this limitation, since Terraform has requested confirmation there's no reason why Terraform couldn't avoid this limitation by listing & deleting any files within the Directory when we go to delete it

This is the solution azurerm users need. The current alternative for us at the moment is https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax#provisioners-are-a-last-resort

mostafaSbakr commented 1 year ago

Using a provisioner is the way to do it for the time being, as @razgrim says. 

  resource "local_file" "script_template" {
      content = templatefile("${path.module}/clear-directory.tpl", {
          storage_account_key    = var.storage_account_key
      })
      filename = "${path.module}/clear-directory.sh"
}

resource "azurerm_storage_share_directory" "directory_per_feature" {
  name                 = var.name
  share_name           = var.fileshare_name
  storage_account_name = var.storage_account_name

  provisioner "local-exec" {
    command     = "${path.module}/clear-directory.sh"
    interpreter = ["bash"]
    when        = destroy

    environment = {
      STORAGE_ACCOUNT_NAME = self.storage_account_name
      FILESHARE_NAME       = self.share_name
      NAME                          = self.name
    }
  }
}

clear-directory.tpl

#!/bin/bash

files=$(az storage file list --account-name "$STORAGE_ACCOUNT_NAME" --account-key ${storage_account_key} --share-name $FILESHARE_NAME --path "$NAME" --query "[].name" --output tsv)

files=$(eval echo $files)
list=( $files )

for file in "$${list[@]}"; do
  az storage file delete --account-name "$STORAGE_ACCOUNT_NAME" --account-key ${storage_account_key} --share-name "$FILESHARE_NAME" --path "$NAME"/"$file"
done
razgrim commented 1 year ago

@mostafaSbakr 's workaround is nice, unfortunately it doesn't work with any ephemeral runner based terraform pipelines (TFC, TFE, Github actions) since the clear directory script won't exist, when the destroy operation that calls it is attempted.

The workaround we had to go with, is to simply not use this azurerm resource at all, create it using a local-exec and let deletion be skipped during destroy operations. 

resource "null_resource" "file_share_dir" {
  provisioner "local-exec" {
    command = <<EOF
      az storage directory create --name $DIR_NAME --share-name $SHARE_NAME --account-name $ACCOUNT_NAME --account-key $ACCOUNT_KEY
    EOF
    environment = {
      DIR_NAME      = dirname
      SHARE_NAME    = yourshare
      ACCOUNT_NAME  = your_storage_account_name
      ACCOUNT_KEY   = your_storage_account_key
    }
  }
}

Understanding that this still doesn't actually solve the "can't delete the directory when there's stuff in it" issue, but it allows the share/storageaccount deletion to not be blocked by a directory, which works for what we're doing.

don't use azurerm_storage_share_directory if you have ephemeral terraform runners, intend on storing unmanaged files in the directory and need to be able to delete the resource. It won't work for you in this scenario, and will require significant workarounds even if you don't use ephemeral runners.

mostafaSbakr commented 1 year ago

@razgrim is correct, running a destroy provisioner with a local_file sometimes succeeds and sometimes fails depending on the order of execution, if the file gets created before the destroy operation takes place it succeeds and fails otherwise (No such file or directory) and there is no way to ensure that the file gets created first using Terraform.

There is no way to configure the order of destroy/apply in Terraform because in this case [depends_on] simply does not work since the required action is to create a script before destroying a share directory, and [depends_on] controls the order of apply or destroy but not both.

To work around this, I utilized Terragrunt dependencies, since I use Terragrunt in my project. That is, modularization. One module will only create a script and another will manage the share directory (and other resources). The module that manages the share directory has a dependency on the first. This ensures that the script is always there for the local_exec during the destroy operation, and will be ignored during apply.

This solution works locally and also in GitLab/Github/etc pipelines since the script will get created each pipeline run, and discarded after the pipeline finishes. And only takes seconds to create.

ffischer1984 commented 1 year ago

@mostafaSbakr terragrunt is totally new for me. Maybe you want to publish an simple example on github/medium.org. I have the same problem but currently I don't have the imagination for this solution or it's just too late in the evening ;)

manicminer commented 7 months ago

Although this issue is very long standing, given that we've recently made improvements to the azurerm_storage_share_file, azurerm_storage_share_directory and azurerm_storage_share_file resources and moved those resources over to our new SDK, I thought I'd give an update here and a short explanation for why we probably won't implement a force-delete option for shares or directories.

When destroying a directory, any files, directories, and nested files/directories, within that directory, that are under management by Terraform, will be destroyed in a deterministic order. Additionally, an resulting eventual consistency issues that arise from rapid deletion of a directory's contents followed by the directory itself, are now automatically handled by the provider. This should result in a clean destroy run when all nested files/directories in a share are under management.

Unfortunately, the API does not expose a force-delete option when deleting shares or directories. If it did, it would be a straightforward addition to the provider to expose this as a feature flag (much as we do something similar for resource groups). However, in the absence of such an option, in order to equip the provider with a force-delete flag for shares/directories, we'd need to implement a recursive deletion algorithm - effectively an implementation of rm -rf for file shares. This is likely to be both dangerous, and also fraught with race conditions reducing its usefulness. In some ways would also go against design principles of Terraform, and so we have elected not to do this, and instead recommend alternative tooling for such a scenario. This can be plugged in as a destroy provisioner, just as some commenters have already done.

Accordingly, since due to our recent updates, we now handle consistency and dependency issues when destroying directories/files under management, I'm going to go ahead and close out this issue as this time. If in future the API adds support for force-deleting directories, we can take another look at this. Thanks!

github-actions[bot] commented 6 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.