Get-AzStorageBlob blocks console in interactive mode if an error ocurrs

[david-garcia-garcia]

Description

I am investigating an unhandled exception in Get-AzStorageBlob. When the error happens, the console gets blocked and you can only regain console control using ctrl+c.

Issue script & Debug output

$ctx = New-AzStorageContext -StorageAccountName $backupUrl.storageAccountName -SasToken $backupUrl.sasToken;

$blobs = Get-AzStorageBlob -Container $backupUrl.container -Context $ctx -Prefix $backupUrl.prefix;

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.20348.2227
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.20348.2227
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module versions

PS C:\> Get-Module Az*

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     2.17.0     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear...
Script     6.1.3      Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountMa...

Error output

PS C:\> Resolve-AzError

   HistoryId: 4

Message        : Method not found: 'Azure.Pageable`1<Azure.Storage.Blobs.Models.BlobItem>
                 Azure.Storage.Blobs.BlobContainerClient.GetBlobs(Azure.Storage.Blobs.Models.BlobTraits,
                 Azure.Storage.Blobs.Models.BlobStates, System.String, System.Threading.CancellationToken)'.
StackTrace     :    at Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobCommand.<ListBlobsByPrefi
                 x>d__52.MoveNext()
                    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine&
                 stateMachine)
                    at Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobCommand.ListBlobsByPrefix
                 (Int64 taskId, IStorageBlobManagement localChannel, String containerName, String prefix, Func`2
                 blobFilter, Boolean includeDeleted, Boolean includeVersion)
                    at Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobCommand.<>c__DisplayClass
                 55_1.<ExecuteCmdlet>b__0(Int64 taskId)
                    at Microsoft.WindowsAzure.Commands.Storage.Common.LimitedConcurrencyTaskScheduler.RunTask(Func`2
                 taskGenerator)
                    at Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobCommand.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : System.MissingMethodException
InvocationInfo : {Get-AzStorageBlob}
Line           :         $blobs = Get-AzStorageBlob -Container $backupUrl.container -Context $ctx -Prefix
                 $backupUrl.prefix;

Position       : At C:\Program Files\WindowsPowerShell\Modules\Sbs\Functions\SbsMssqlPrepareRestoreFiles.ps1:22 char:18
                 + ...    $blobs = Get-AzStorageBlob -Container $backupUrl.container -Contex ...
                 +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 4

[blueww]

@isra-fel

Would you please help to look at this issue?

It looks all Storage dataplane cmdlets based on v12 SDK (Track2 SDK) won't work, when customer first run cmdlet "Connect-DbaInstance" (from module "dbatools"), then run storage dataplane cmdlet like "Get-AzStorageContainer" or "Get-AzStorageBlob". Not sure if cmdlets in other modules based on track2 SDK will be impacted or not.

I can repro this issue with following script in both Powershell 5/7:

Connect-DbaInstance

$ctx = New-AzStorageContext -StorageAccountName $accountname -StorageAccountKey $key
$blobs = Get-AzStorageBlob -Container $containername -Context $ctx -Prefix "xxx"

A workaround might be first run "import-module Az.Storage", then run "Connect-DbaInstance". On my machine, looks this way won't fail any cmdlet.

[blueww]

Update: From https://github.com/dataplat/dbatools/issues/9335, it looks might related with Azure Identity in "dbatools" is lower than the one used by Azure PowerShell modules.

@isra-fel Would you please help to see is there any way we can do to resolve this issue?

[david-garcia-garcia]

Just to clarify, although this is a version mismatch and can be worked-around adjusting the import order, there is a secondary issue "blocks console in interactive mode". So, it's not just that an error happens, it's that the command which should run unatended is not releasing the console but blocking the whole process. There is something wrong with error handling in this library.

[blueww]

@isra-fel

Do you have any idea to improve the error handling to avoid cmdlet hang for above assembly conflict error?

@david-garcia-garcia I believe you are not blocked for this issue now. There might be another way to avoid script blocking for a single cmdlet hang, you might can run the cmdlet as a backend job to avoid it blocks the whole script, see https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/start-job?view=powershell-7.4

[david-garcia-garcia]

@blueww current error handling behaviour (process gets stuck/ransomed) is super dangerous.

How am I supposed to know that the process has en error if it does not return or throw an exception?

I am making these calls from a docker entrypoint and from scheduled tasks. Current behaviour is totally unreliable, nothing can be captured o handled because the process does not return.

I'm OK if it fails due to version mismatch, but it has to fail, not to block indefinately.

[blueww]

@david-garcia-garcia

We can understand your concern for the cmdlet not exit in this kind of error.

The cmdlets can exist on common type of error, like error reported from storage server, like wrong parameters are input. However, for this kind of error, it's caused by assembly conflict, it looks the logic is not in Storage cmdlet logic, but related with Azure PowerShell framework for handle assembly conflict. @isra-fel, do you have any suggestion how to avoid cmdlet hang on this kind of error?