Cannot delete full directory structure from Directory in File Share

[dsparkplug]

Is your feature request related to a problem? Please describe.

The az storage file upload-batch command can be used to upload a full directory structure to an Azure File Share or Directory.

The az storage file delete-batch command can be used to recursively delete all the files in an Azure File Share or Directory. It does however leave all the empty subfolders.

There does not seem to be a way of deleting all the empty subfolders - unless each subfolder is deleted individually. This requires prior knowledge of the folder names.

Describe the solution you'd like

The az storage file delete-batch command should have an option to delete empty subfolders. Alternatively, or additionally, the az storage directory delete should have an option to delete all subfolders.

Describe alternatives you've considered

Note that is is possible to delete the whole file share using az storage share delete --name sparkysfileshare just not a specific directory within a file share.

Additional context

Here's an example workflow:

> az storage directory create --name sparkysfolder --share-name sparkysfileshare 
{
  "created": true
}

> az storage file upload-batch --destination sparkysfileshare --source I:\temp\test --destination-path sparkysfolder --pattern * 

uploading I:\temp\test\test1.txt
Finished[#############################################################]  100.0000%
uploading I:\temp\test\subfolder\test2.txt
Finished[#############################################################]  100.0000%
[
  "https://vsoftfilestorageac.file.core.windows.net/sparkysfileshare/sparkysfolder/test1.txt",
  "https://vsoftfilestorageac.file.core.windows.net/sparkysfileshare/sparkysfolder/subfolder/test2.txt"
]

> az storage file list --share-name sparkysfileshare --path sparkysfolder

[
  {
    "metadata": null,
    "name": "test1.txt",
    "properties": {
      "contentLength": 4,
      "contentRange": null,
      "contentSettings": {
        "cacheControl": null,
        "contentDisposition": null,
        "contentEncoding": null,
        "contentLanguage": null,
        "contentMd5": null,
        "contentType": null
      },
      "copy": {
        "completionTime": null,
        "id": null,
        "progress": null,
        "source": null,
        "status": null,
        "statusDescription": null
      },
      "etag": null,
      "lastModified": null,
      "serverEncrypted": null
    },
    "type": "file"
  },
  {
    "metadata": null,
    "name": "subfolder",
    "properties": {
      "etag": null,
      "lastModified": null,
      "serverEncrypted": null
    },
    "type": "dir"
  }
]

> az storage file delete-batch --source sparkysfileshare --pattern sparkysfolder\* --account-name vsoftfilestorageac 

> az storage file list --share-name sparkysfileshare --path sparkysfolder

[
  {
    "metadata": null,
    "name": "subfolder",
    "properties": {
      "etag": null,
      "lastModified": null,
      "serverEncrypted": null
    },
    "type": "dir"
  }
]

> az storage directory delete --name sparkysfolder --share-name sparkysfileshare 

The command failed with an unexpected error. Here is the traceback:

The specified directory is not empty. ErrorCode: DirectoryNotEmpty
<?xml version="1.0" encoding="utf-8"?><Error><Code>DirectoryNotEmpty</Code><Message>The specified directory is not empty.
RequestId:17d1cf8b-001a-000b-7fab-f584e4000000
Time:2019-04-18T05:59:17.7350008Z</Message></Error>
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\knack\knack\cli.py", line 206, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 326, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 384, in _run_jobs_serially
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 377, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\six\six.py", line 693, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 354, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 145, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\__init__.py", line 451, in default_command_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\file\fileservice.py", line 1035, in delete_directory
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\_error.py", line 97, in _dont_fail_not_exist
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\file\fileservice.py", line 1032, in delete_directory
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\storageclient.py", line 381, in _perform_request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\storageclient.py", line 306, in _perform_request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\storageclient.py", line 292, in _perform_request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\_error.py", line 115, in _http_error_handler
azure.common.AzureConflictHttpError: The specified directory is not empty. ErrorCode: DirectoryNotEmpty
<?xml version="1.0" encoding="utf-8"?><Error><Code>DirectoryNotEmpty</Code><Message>The specified directory is not empty.
RequestId:17d1cf8b-001a-000b-7fab-f584e4000000
Time:2019-04-18T05:59:17.7350008Z</Message></Error>

[limingu]

We will add the option to 'az storage directory delete' to delete all empty subfolders.

[ryansantaana]

Is this feature going to be added soon? I have a CI/CD workflow within Azure Pipelines where I need to delete a directory and upload a new one with each release, and this feature would be nice to have.

[Juliehzl]

@dsparkplug You need to use az storage directory delete --name sparkysfolder/subfolder --share-name sparkysfileshare to delete empty folder. But we will improve it in S162.

[dsparkplug]

@Juliehzl

@dsparkplug You need to use az storage directory delete --name sparkysfolder/subfolder --share-name sparkysfileshare to delete empty folder. But we will improve it in S162.

If you look at my example workflow above, you will see that is one of the commands that I did use. The issue was not with deleting a single empty folder - the issue was with deleting a full directory structure.

I look forward to seeing the improvements in S162

[ekelmans]

you gotta be kidding me....

So i have a tree with 40.000+ folders and over 3.000.000 files, and those need to be deleted on a per item basis ?

[ChandraSM]

Modified version of the above script to delete a sub-directory under main directory inside a file share:

function RemoveFileDir ([Microsoft.Azure.Storage.File.CloudFileDirectory] $dir, [Microsoft.Azure.Commands.Common.Authentication.Abstractions.IStorageContext] $ctx) {
$filelist = Get-AzStorageFile -Directory $dir foreach ($f in $filelist) {
if ($f.GetType().Name -eq "CloudFileDirectory") { RemoveFileDir $f $ctx #Calling the same unction again. This is recursion. } else { Remove-AzStorageFile -File $f
} } Remove-AzStorageDirectory -Directory $dir }

define varibales

$StorageAccountName = "storageaccountname" $StorageAccountKey = "storageaccountkey" $AzShare = "filesharename" $AzDirectory = "rootdirectoryunderfileshare"

create primary region storage context

$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey $ctx.ToString()

Check for Share Existence

$S = Get-AzStorageShare -Context $ctx -ErrorAction SilentlyContinue|Where-Object {$_.Name -eq $AzShare}

Check for directory and delete the sub directory which is not needed-

$d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name

if ($d.Name -notcontains $AzDirectory) { Write-Host "directory is not present; no action to be performed" } else {

Write-Host "Entered"

$dir1 = Get-AzStorageFile -Share $S -Path $AzDirectory
$dir2 = Get-AzStorageFile -Directory $dir1 | Where-Object Name -EQ "subdirectoryunderroot"
$dir3 = Get-AzStorageFile -Directory $dir2 | Where-Object Name -EQ "foldertodeleteundersubdirectory"
RemoveFileDir $dir3 $ctx   

}

[zhoxing-ms]

I think it is necessary for the Service to implement the function of deleting the sub empty folder, the main reasons are as follows:

1. If the CLI implements the deletion of all subfolders by itself, it will recursively query and delete all subdirectories. If the number of folders is large, the performance will be slow due to too many requests, and if the folder is deep, the stack will overflow
2. Customers who invoke the rest API may also have a need to delete all empty subfolders under a folder.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[CondormanFr]

The command : $d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7 It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

.

[BryanSoltis]

The command : $d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7 It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

.

I'm getting this same error when developing a PowerShell Azure Function. Any update on this issue?

[ekelmans]

Nope,

No reply from MS on this issue yet ☹

Met vriendelijke groet,

Theo Ekelmans Sr. MS-SQL DBA

Ringwade 1 3439 LM Nieuwegein T +3130 663 7000 M +316 2127 4593 http://www.ordina.nl/ www.ordina.nl

Eerstelijn MS SQL Team Werkdagen tussen 08:00 – 18:00 T +3130 663 77 77 E mailto:mssql@ordina.nl mssql@ordina.nl

7x24 klanten: 0800-0231841

From: Bryan Soltis notifications@github.com Sent: Monday, 22 June 2020 17:03 To: Azure/azure-cli azure-cli@noreply.github.com Cc: Theo Ekelmans theo@ekelmans.com; Comment comment@noreply.github.com Subject: Re: [Azure/azure-cli] Cannot delete full directory structure from Directory in File Share (#9141)

The command : $d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7 It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

.

I'm getting this same error when developing a PowerShell Azure Function. Any update on this issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cli/issues/9141#issuecomment-647575028 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIBPH4VJBDQQ5TWUZL5MMJTRX5XC5ANCNFSM4HG2AFYQ . https://github.com/notifications/beacon/AIBPH4VHV3G3QXTRWYAMX5LRX5XC5A5CNFSM4HG2AFY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOE2MTL5A.gif

[edwardwu-servian]

you gotta be kidding me....

So i have a tree with 40.000+ folders and over 3.000.000 files, and those need to be deleted on a per item basis ?

@ekelmans try az storage file delete-batch -s=$SHARE_NAME --account-name=$ACCOUNT_NAME. i've recently bump into issue with az storage directory delete (I am using terraform - under the hood it's using Azure REST API )

[janegilring]

@ekelmans I currently experience the same when developing an Azure Function running PowerShell 7 for exporting some files to an Azure File Share. Did you have any progress on your case?

Update: The following alternate technique seems to work

$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $BackupStorageAccountResourceGroupName -Name $BackupStorageAccountName)[0].Value
$Context = New-AzStorageContext -StorageAccountName $BackupStorageAccountName -StorageAccountKey $StorageAccountKey

Get-ChildItem -Recurse -File -Path Temp:/backup/ | ForEach-Object {
    $path = '/Backups/KeyVault/' + (Split-Path $_.FullName -Leaf)
    Set-AzStorageFileContent -ShareName $BackupStorageAccountFileShareName -Source $_.FullName -Path $path -Force -Context $Context
    $path
}

[ekelmans]

@ekelmans I currently experience the same when developing an Azure Function running PowerShell 7 for exporting some files to an Azure File Share. Did you have any progress on your case?

Update: The following alternate technique seems to work

$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $BackupStorageAccountResourceGroupName -Name $BackupStorageAccountName)[0].Value
$Context = New-AzStorageContext -StorageAccountName $BackupStorageAccountName -StorageAccountKey $StorageAccountKey

Get-ChildItem -Recurse -File -Path Temp:/backup/ | ForEach-Object {
    $path = '/Backups/KeyVault/' + (Split-Path $_.FullName -Leaf)
    Set-AzStorageFileContent -ShareName $BackupStorageAccountFileShareName -Source $_.FullName -Path $path -Force -Context $Context
    $path
}

Hi J,

No answer just yet, though the dev team said they have added it to the todo list, but gave no timeframe for a new release.

The problem with the foreach-object loop is that is is Very slow if you have millions of files in thousands of folders. BUT... for now it helps to get rid of the smallest folders

Thanks fo sharing :)

[janegilring]

I see. Did you try the new -Parallel switch on Foreach-Object in PowerShell 7?

[ekelmans]

WOW.... No,

I never knew the existence of such a switch, I will give it a try, thanks again :)

[bpossolo]

+1 please add a way to delete empty folder tree using the azure-cli or better yet, update az storage file delete-batch to delete the folders as well.... that's what i expected it would do

[zffocussss]

you gotta be kidding me....

So i have a tree with 40.000+ folders and over 3.000.000 files, and those need to be deleted on a per item basis ?

so disgusting if specifying per item....

[bpossolo]

@zffocussss you can delete files in bulk using the command I wrote... the issue is the folders don't get deleted..

[tmasabari]

you gotta be kidding me.... So i have a tree with 40.000+ folders and over 3.000.000 files, and those need to be deleted on a per item basis ?

so disgusting if specifying per item....

I am also facing a similar issue. If file share is mapped as the Windows drive and the user performed a similar delete operation, does Windows delete each item separately ?!!! (even if it is a parallel operation it is very time consuming) There should be an inbuilt call to delete the folders recursively.

Interestingly blob container with Azure Data Lake Gen 2 enabled supports this recursive folder delete operation !!!

[dazinator]

This still hasn't been addressed.. Please make it easy to delete n entire directory - in windows explorer you can right click on a folder and click delete and confirm you want to delete recursively all contents. Simple. This would translate to something like:

az storage directory delete --name MyFolder --share-name myshare --recursive

Not leaving behind empty directories when you delete a batch of files using a glob patterns, seems like a seperate use case which would also need some new flag

az storage file delete-batch -s=$SHARE_NAME --account-name=$ACCOUNT_NAME --pattern foo\* --remove-empty-directories

[joelmforsyth]

The command : $d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7 It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

The property $S.CloudFileShare is a Microsoft.Azure.Storage.File.CloudFileShare. Have you tried just updating the call to be $d = Get-AzStorageFile -Share $S.CloudFileShare -ErrorAction SilentlyContinue|select Name?

This solved the issue for me.

[navba-MSFT]

@dsparkplug Apologies for the late reply. We will provide an update on this once we have more details on this.

@zhoxing-ms @Juliehzl Could you please provide an update on this once you get a chance ? Awaiting your reply.

[evelyn-ys]

Storage service has to support first @navba-MSFT

[kvskranthikumar]

Any update?

[bpossolo]

looks like this is supported now 🎉 I tried it from Microsoft Azure Storage Explorer and it successfully deleted all empty folders recursively. If you want to do it from the command-line with azcopy, this is the command Azure Storage Explorer used:

azcopy remove "https://<account>.file.core.windows.net/<fileshare>/<directory>?<sas-string>" --from-to=FileTrash --recursive

[ekelmans]

looks like this is supported now 🎉 I tried it from Microsoft Azure Storage Explorer and it successfully deleted all empty folders recursively. If you want to do it from the command-line with azcopy, this is the command Azure Storage Explorer used:

azcopy remove "https://<account>.file.core.windows.net/<fileshare>/<directory>?<sas-string>" --from-to=FileTrash --recursive

YESSS !!!

Finally !

[Matthew0x]

I tried implementing recursive file removal using the dedicated AZ CLI module and I can't receive any parsable command output from the az storage directory delete

The entire command isn't respected in try/catch (it returns an error, but that error isn't an object that I would intercept to a variable. Other commands usually work fine) and thus I can't research the state of directories on the network file share (via Azure CLI API response).

If you protect your API from spamming the endpoint, then at least please provide an alternative that works with the dedicated toolset... (azure cli) on your side. That azcopy doesn't seem to accept connection strings. There is a huge mess in the commands' structure, since the parameters aren't standardized and uniform across seemingly even the same modules.

Examples: az storage directory exists --name (it's a path) az storage directory show/delete --name (it's a path)

az storage file delete --path (it's a path) az storage file list --path (it's a path)

az storage file delete-batch --source (it's a path)

What is going on in here? Who wrote this? So now instead of using Azure Cli on agents I am supposed to install azcopy then cut connection strings into SAS, then create own URL endpoints to utilize the recursive directory removal?

I have no words.

[silverl]

Here's my contribution. A PowerShell Core script using Az.Storage that deletes File Share folders and files older than a specified date, recursively.

Make sure you've logged in (Connect-AzAccount) first. You can use this with Managed Identity on a VM by embedding the Connect-AzAccount call in the script, specifying parameters for -SubscriptionId -Identity and -AccountId (use the Managed Identity's ClientId GUID).

You can enhance with ForEach-Object -ThrottleLimit # -Parallel but you have to be careful to handle out-of-scope variables or functions.

param(
    [string] $DirectoryPath,
    [string] $StorageAccountName,
    [string] $FileShareName,
    [string] $ResourceGroupName,
    [int] $DaysOld
)

Set-StrictMode -Version Latest

$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"

# Get the current date
$CurrentDate = Get-Date

# Get the date to delete files older than
$DateToDelete = $CurrentDate.AddDays(-$DaysOld)

$StorageAccount = Get-AzStorageAccount -Name $StorageAccountName -ResourceGroupName $ResourceGroupName
# Create a storage context
$StorageContext = $StorageAccount.Context

# Define a function to recursively delete files in a directory
function Remove-FilesRecursively {
    # Get the directory path from the first argument
    $DirectoryPath = $Args[0]
    Write-Output "Entering $DirectoryPath"

    # Get all the files and directories in the directory
    Get-AzStorageFile -ShareName $FileShareName -Path $DirectoryPath -Context $StorageContext | Get-AzStorageFile `
    | ForEach-Object {
        $Item = $_
        $Item.Name
        # If the item is a file, check if it is older than the date to delete and remove it if true
        if ($Item.GetType().Name -eq "AzureStorageFile" -and $Item.LastModified -lt $DateToDelete) {
            Write-Output "Removing $($Item.Name)"
            $Item | Remove-AzStorageFile
        }
        # If the item is a directory, call this function recursively with its path as an argument
        elseif ($Item.GetType().Name -eq "AzureStorageFileDirectory") {
            Write-Output "Going into /$($DirectoryPath)/$($Item.Name)"
            Remove-FilesRecursively "$($DirectoryPath)/$($Item.Name)" @PSBoundParameters
        }
    }

    # Check if the directory is empty after deleting the files and directories inside it
    $folder = Get-AzStorageFile -ShareName $FileShareName -Path $DirectoryPath -Context $StorageContext
    $Contents = $folder | Get-AzStorageFile
    if ($null -eq $Contents) {
        Write-output "Removing empty folder $DirectoryPath"
        $folder | Remove-AzStorageDirectory
    }
}

# Call the function with an empty string as an argument to start from the root directory of the file share
Remove-FilesRecursively $DirectoryPath @PSBoundParameters

[Matthew0x]

This script is pretty cool, really.

I wrote something similar at the time, but I believe I tried first removing the directory, then searching for items if that failed (so in reverse, your approach might be better). There were/are also some error handling issues with that Python extension if I recall correctly (that was some time ago)

In my opinion they should just allow the API to accept a directory removal request, then they should recursively remove files inside a directory on their own agent. That would not only save the network bandwith by escaping an API spam, but would also be much faster due to the action taking place on the local OS.

Right now it's pretty slow (at least from my experience).

[silverl]

It's truly, very slow, especially if any folder has hundreds of thousands of files, or millions, as in my case. The list of files gets loaded into RAM and piped into a ForEach-Object loop. I can see the virtual machine virtual memory growing in these cases. I've read that piping a Get- into a ForEach-Object loop can allow for streaming of the results from one into the other, avoiding memory problems.

I have another script I wrote for when you're on a VM with access to the file share using a UNC path beginning with \\servername\sharename\folders. I can share it if you like.

[Matthew0x]

I agree that mounting a File Share on an agent then using something like Python or PowerShell Core could be much faster.

I would have probably tried that at the time, but I lost the chance. I used recursion labels to make it funny. You can use while() with labels and make GO TO "hacks" against the design of the language :rofl:

[silverl]

Here's the one I'm using on a VM with a file share. Uses Parallelism.

It's still slow. Also I get network errors from time to time and had to add a Retry-Command.

param (
    $StartPath = "\\your_storage_account_name.file.core.windows.net\your_share_name\whatever"
)

Set-StrictMode -Version Latest

Start-Transcript -Path "$PSScriptRoot\Purge-OldShareFiles.log" -Force

$stopWatch = [System.Diagnostics.Stopwatch]::StartNew()

Write-Output "Started at $(Get-Date)"

$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"

$foldersToPurge = @(
    [pscustomobject]@{Name = 'Folder1'; RetentionInDays = 180 }
    [pscustomobject]@{Name = 'Folder2'; RetentionInDays = 365 }
    [pscustomobject]@{Name = 'Folder3'; RetentionInDays = 30 }
)

function Retry-Command {
    [CmdletBinding()]
    Param(
        [Parameter(Position = 0, Mandatory = $true)]
        [scriptblock]$ScriptBlock,

        [Parameter(Position = 1, Mandatory = $false)]
        [int]$Maximum = 5,

        [Parameter(Position = 2, Mandatory = $false)]
        [int]$Delay = 100
    )

    Begin {
        $cnt = 0
    }

    Process {
        do {
            $cnt++
            try {
                $ScriptBlock.Invoke()
                return
            }
            catch {
                Write-Error $_.Exception.InnerException.Message -ErrorAction Continue
                Start-Sleep -Milliseconds $Delay
            }
        } while ($cnt -lt $Maximum)

        # Throw an error after $Maximum unsuccessful invocations. Doesn't need
        # a condition, since the function returns upon successful invocation.
        throw 'Execution failed.'
    }
}

$funcDef = ${function:Retry-Command}.ToString()

# Define a recursive function
function Remove-Files ($Path, $Days) {

    Write-Output "Processing folder $($Path)"

    # Get current date
    $CurrentDate = Get-Date

    # Calculate date to delete
    $DateToDelete = $CurrentDate.AddDays(-$Days)

    if (! ( Test-Path -Path $Path) ) {
        continue
    }

    # Delete files in parallel older than date
    Get-ChildItem -Path $Path -File | Where-Object { $_.LastWriteTime -lt $DateToDelete } | ForEach-Object -ThrottleLimit 10 -Parallel {
        $File = $_

        # Bring in out-of-scope function.
        ${function:Retry-Command} = $using:funcDef

        Write-Output "Removing $($File.FullName)"
        Retry-Command -Delay 1000 -ScriptBlock { Remove-Item -Path $File.FullName } -Maximum 5
    }

    # Get folders in current path
    $Folders = Get-ChildItem -Path $Path -Directory -ErrorAction SilentlyContinue

    # Call function for each subfolder
    $Folders | ForEach-Object {
        $Folder = $_
        Remove-Files -Path $Folder.FullName -Days $Days
        # Delete empty folder
        if ($null -eq (Get-ChildItem -Path $Folder.FullName)) {
            Write-Output "Removing empty folder $($Folder.FullName)"
            Remove-Item -Path $Folder.FullName
        }
    }
}

$folders = Get-ChildItem -Directory -Path $StartPath

foreach ($clientFolder in $folders) {
    foreach ($folder in $foldersToPurge) {

        $fullpath = $clientFolder.FullName + "\$($folder.Name)"

        Remove-Files -Path $fullpath -Days $folder.RetentionInDays
    }
}

$stopWatch.Stop()
$stopWatch.Elapsed.ToString()
Write-Output "Ended at $(Get-Date)"

# Stop Logging
Stop-Transcript

[silverl]

I just updated script above for the one I run on the VM to delete from a file share. Piping directly into the Where-Object eliminated the excessive memory usage I was experiencing when dealing with folders with 1M files or more. It allows the list of files to be filtered for date using a stream and avoids loading them all into memory for the for loop.

[gukoff]

For those who want to recursively delete a directory on a file share using az CLI rather than lots of powershell, here's an example script. (it's still powershell, but easily translates to bash or any other shell):

# recursively delete directory "a/b/c" on fileshare "myfileshare" on storage account "mystorageacc"

$storageAccountName = "mystorageacc"
$fileShareName = "myfileshare"
$directoryToDelete = "a/b/c"
$containerResourceGroup = "myResourceGroup"

$mountDirectory = "/mnt/fileshare"

$storageKey = (az storage account keys list --account-name "$storageAccountName" --query "[0].value" --output tsv)

az container create `
  --name 'cleanup' `
  --image 'mcr.microsoft.com/azure-cli:latest' `
  --command-line "rm -r '$mountDirectory/$directoryToDelete'" `
  --restart-policy 'Never' `
  --resource-group $containerResourceGroup `
  --azure-file-volume-account-name $storageAccountName `
  --azure-file-volume-account-key $storageKey `
  --azure-file-volume-share-name $fileShareName `
  --azure-file-volume-mount-path $mountDirectory

Here we mount the file share on an ACI container and remove the directory using the plain rm -r.

Using the image mcr.microsoft.com/azure-cli:latest is not essential, any image with rm will do the trick.

This approach is also immune to the injections or mistakes with the special symbols in directory path such as []?*., unlike the --pattern argument in az storage file delete-batch.

[mloskot]

az storage remove

Remove an entire directory:

az storage remove --account-name "${ASA_NAME}"  --share-name "${SHARE_NAME}" --path abc --recursive

where abc is empty or non-empty directory on Azure File Share.

Remove everything inside Azure Storage File Share:

az storage remove --account-name "${ASA_NAME}"  --share-name "${SHARE_NAME}" --recursive