Support for `recursive` delete of `azurerm_storage_data_lake_gen2_path`

[camilo-s]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Currently, it is not possible to delete a directory inside a hierarchical namespace-enabled container (a.k.a. Data Lake Filesystem paths) with Terraform if it contains data.

Terraform returns the following error at the apply stage:

│ Error: deleting Path "###" in File System "###" in Storage Account "###": datalakestore.Client#Delete: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="DirectoryNotEmpty" Message="The recursive query parameter value must be true to delete a non-empty directory.\nRequestId:###\nTime:###"

In the Azure Portal, it's possible to do so, and the documentation for the Path Delete REST API states that a recursive parameter can be passed in the URI to allow the deletion of directories and their content: https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete#uri-parameters

Without this feature, it's not possible to completely manage filesystem paths via Terraform.

New or Affected Resource(s)/Data Source(s)

azurerm Provider features, azurerm_storage_data_lake_gen2_path

Potential Terraform Configuration

provider "azurerm" {
  features {
    storage_account {
      enable_recursive_filesystem_path_delete = true
    }
  }
}

References

No response

[seblatre]

For those who are really annoyed by this limitation, I have successfully implemented the following workaround (hack?) that requires bash, tr and Azure cli available:

# Resource to full destroy including its content
resource "azurerm_storage_data_lake_gen2_path" "adl2" {
   ...
}
# Having the triggers referencing the resource to delete is necessary to have the right deletion order.
# If not the case, use a depends_on directive to the "adl2" resource
resource "null_resource" "folder_destroy" {
  triggers = {
    storage_account_name = var.storage_account.name
    filesystem           = azurerm_storage_data_lake_gen2_path.adl2.filesystem_name
    folder_path          = azurerm_storage_data_lake_gen2_path.adl2.path
  }

  provisioner "local-exec" {
    when        = destroy
    command     = <<-EOT
      az storage fs file list \
        --auth-mode login \
        --account-name '${self.triggers.storage_account_name}' \
        --file-system '${self.triggers.filesystem}' \
        --path '${self.triggers.folder_path}' \
        --recursive false \
        --output tsv \
        --query '[].name' | \
      tr -d '\r' | \
      while read to_delete_name ; do 
        echo "Removing item $to_delete_name from folder ${self.triggers.folder_path}"
        az storage fs directory delete \
        --auth-mode login \
        --yes \
        --account-name '${self.triggers.storage_account_name}' \
        --file-system '${self.triggers.filesystem}' \
        --name "$to_delete_name"
      done
    EOT
    working_dir = path.module
    interpreter = ["bash", "-c"]
  }
}