search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.26k stars 3.86k forks source link

Run a cmdlet with -Asjob, when error report in EndProcessing(), the job state will still be "Completed" #13105

Open blueww opened 4 years ago

blueww commented 4 years ago

Description

Run a cmdlet with -Asjob, when error report in EndProcessing(), the job state will still be "Completed". This will misleading customer think the cmdlet runs success, but actually failed.

Steps to reproduce

Try to overwrite a block blob with page blob, If you run it with -Asjob, will see the task finish with State as "Completed" ($task.error is not null). But if you run it without -asjob, will see error reported.

This is caused by the failure happens in EndProcessing()

with -asjob
PS C:\temp> $task = Set-AzStorageBlobContent -File .\testfile_2048K -Container $containerName -Blob test.txt -Force -Context $ctx -BlobType Page -AsJob

PS C:\temp> $task

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command                  
--     ----            -------------   -----         -----------     --------             -------                  
2      Long Running... AzureLongRun... Completed     True            localhost            Set-AzStorageBlobContent 

PS C:\temp> $task.Error
The blob type is invalid for this operation. HTTP Status Code: 409 - HTTP Error Message: The blob type is invalid for this operation.
ErrorCode: InvalidBlobType
ErrorMessage: The blob type is invalid for this operation.
RequestId:13e87fc5-401e-001a-6815-96bb1d000000
Time:2020-09-29T04:04:15.1951577Z
    + CategoryInfo          : CloseError: (:) [], StorageException
    + FullyQualifiedErrorId : StorageException
without -asjob
PS C:\temp> Set-AzStorageBlobContent -File .\testfile_2048K -Container $containerName -Blob test.txt -Force -Context $ctx -BlobType Page
Set-AzStorageBlobContent : The blob type is invalid for this operation. HTTP Status Code: 409 - HTTP Error Message: The blob type is invalid for this operation.
ErrorCode: InvalidBlobType
ErrorMessage: The blob type is invalid for this operation.
RequestId:1ce5ce3f-a01e-0070-0d11-966335000000
Time:2020-09-29T03:34:39.9380626Z
At line:1 char:1
+ Set-AzStorageBlobContent -File .\testfile_2048K -Container $container ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Set-AzStorageBlobContent], StorageException
    + FullyQualifiedErrorId : StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand

Environment data

PS C:\temp> $PSVersionTable

Name                           Value                                                                                                                                                                                                                                                                                                     
----                           -----                                                                                                                                                                                                                                                                                                     
PSVersion                      5.1.19041.1                                                                                                                                                                                                                                                                                               
PSEdition                      Desktop                                                                                                                                                                                                                                                                                                   
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                                                                                                   
BuildVersion                   10.0.19041.1                                                                                                                                                                                                                                                                                              
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                                                                                           
WSManStackVersion              3.0                                                                                                                                                                                                                                                                                                       
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                                                                                                       
SerializationVersion           1.1.0.1

Module versions

This happens on lastest Az.Storage 2.6.0 + Az.Account 1.9.4


PS C:\temp> Get-Module  

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                                                                                               
---------- -------    ----                                ----------------                                                                                                                                                                                                                                                               
Script     1.9.4      Az.Accounts                         {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount...}                                                                                                                                                                                                    
Script     2.6.0      Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetworkRule, Close-AzStorageFileHandle...}                                                                                                                                 
Script     1.0.0.0    ISE                                 {Get-IseSnippet, Import-IseSnippet, New-IseSnippet}                                                                                                                                                                                                                            
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}                                                                                                                                                                                                             
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {ConvertFrom-SecureString, ConvertTo-SecureString, Get-Acl, Get-AuthenticodeSignature...}                                                                                                                                                                                      
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object...}                                                                                                                                                                                                                      
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Connect-WSMan, Disable-WSManCredSSP, Disconnect-WSMan, Enable-WSManCredSSP...}                                                                                                                                                                                                
Binary     1.1.0.0    PSScheduledJob                      {Add-JobTrigger, Disable-JobTrigger, Disable-ScheduledJob, Enable-JobTrigger...}                                                                                                                                                                                               
Manifest   2.0.0.0    PSWorkflow                          {New-PSWorkflowExecutionOption, New-PSWorkflowSession, nwsn}

Debug output

Add -debug , the result is same.

PS C:\temp> $task = Set-AzStorageBlobContent -File .\testfile_2048K -Container $containerName -Blob test.txt -Force -Context $ctx -BlobType Page -AsJob -Debug
DEBUG: 12:08:21 PM - Use storage account 'weirp1' from storage context.

PS C:\temp> $task

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command                  
--     ----            -------------   -----         -----------     --------             -------                  
3      Long Running... AzureLongRun... Completed     True            localhost            Set-AzStorageBlobContent 

PS C:\temp> $task.Error
The blob type is invalid for this operation. HTTP Status Code: 409 - HTTP Error Message: The blob type is invalid for this operation.
ErrorCode: InvalidBlobType
ErrorMessage: The blob type is invalid for this operation.
RequestId:f54b558d-801e-0005-2c16-960819000000
Time:2020-09-29T04:08:26.0895834Z
    + CategoryInfo          : CloseError: (:) [], StorageException
    + FullyQualifiedErrorId : StorageException

Error output

PS C:\temp> Resolve-AzError
WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

   HistoryId: 6

Message        : The blob type is invalid for this operation. HTTP Status Code: 409 - HTTP Error Message: The blob type is invalid for this operation.
                 ErrorCode: InvalidBlobType
                 ErrorMessage: The blob type is invalid for this operation.
                 RequestId:1ce5ce3f-a01e-0070-0d11-966335000000
                 Time:2020-09-29T03:34:39.9380626Z
StackTrace     : 
Exception      : Microsoft.Azure.Storage.StorageException
InvocationInfo : {Set-AzStorageBlobContent}
Line           : Set-AzStorageBlobContent -File .\testfile_2048K -Container $containerName -Blob test.txt -Force -Context $ctx -BlobType Page
Position       : At line:1 char:1
                 + Set-AzStorageBlobContent -File .\testfile_2048K -Container $container ...
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 6

Message        : The blob type is invalid for this operation.
StackTrace     :    at Microsoft.WindowsAzure.Commands.Storage.Common.DataMovementTransferHelper.<DoTransfer>d__0.MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand.<Upload2Blob>d__58.MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at Microsoft.WindowsAzure.Commands.Storage.Common.LimitedConcurrencyTaskScheduler.<RunConcurrentTask>d__26.MoveNext()
Exception      : Microsoft.Azure.Storage.StorageException
InvocationInfo : {Set-AzStorageBlobContent}
Line           : Set-AzStorageBlobContent -File .\testfile_2048K -Container $containerName -Blob test.txt -Force -Context $ctx -BlobType Page
Position       : At line:1 char:1
                 + Set-AzStorageBlobContent -File .\testfile_2048K -Container $container ...
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 6

   HistoryId: 5

Message        : This operation is only supported on a hierarchical namespace account.
StackTrace     :    at Azure.Storage.Files.DataLake.DataLakeRestClient.Path.GetPropertiesAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response)
                    at Azure.Storage.Files.DataLake.DataLakeRestClient.Path.<GetPropertiesAsync>d__12.MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at Azure.Storage.Files.DataLake.DataLakePathClient.<GetAccessControlInternal>d__67.MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at Azure.Storage.Files.DataLake.DataLakePathClient.GetAccessControl(Nullable`1 userPrincipalName, DataLakeRequestConditions conditions, CancellationToken cancellationToken)
                    at Azure.Storage.Files.DataLake.DataLakeFileClient.GetAccessControl(Nullable`1 userPrincipalName, DataLakeRequestConditions conditions, CancellationToken cancellationToken)
                    at Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureDataLakeGen2Item..ctor(DataLakeFileClient fileClient)
                    at Microsoft.WindowsAzure.Commands.Storage.StorageCloudBlobCmdletBase.WriteDataLakeGen2Item(IStorageBlobManagement channel, DataLakeFileClient fileClient, Nullable`1 taskId)
                    at Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.NewAzDataLakeGen2ItemCommand.<Upload2Blob>d__55.MoveNext()
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                    at Microsoft.WindowsAzure.Commands.Storage.Common.LimitedConcurrencyTaskScheduler.<RunConcurrentTask>d__26.MoveNext()
Exception      : Azure.RequestFailedException
InvocationInfo : {New-AzDataLakeGen2Item}
Line           : New-AzDataLakeGen2Item -FileSystem $containerName -Path testfile -Source C:\temp\1.txt -Context $ctx
Position       : At line:1 char:1
                 + New-AzDataLakeGen2Item -FileSystem $containerName -Path testfile -Sou ...
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 5

Message        : Exception of type 'System.Exception' was thrown.
StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzDataLakeGen2Item}
Line           : New-AzDataLakeGen2Item -FileSystem $containerName -Path testfile -Source C:\temp\1.txt -Context $ctx
Position       : At line:1 char:1
                 + New-AzDataLakeGen2Item -FileSystem $containerName -Path testfile -Sou ...
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 5

   HistoryId: -1

Message        : You cannot call a method on a null-valued expression.
StackTrace     :    at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
                    at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
                    at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockClauseToInvoke clauseToInvoke, Boolean createLocalScope, Dictionary`2 functionsToDefine, List`1 variablesToDefine, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, 
                 InvocationInfo invocationInfo, Object[] args)
                    at System.Management.Automation.ScriptBlock.<>c__DisplayClass57_0.<InvokeWithPipe>b__0()
                    at System.Management.Automation.Runspaces.RunspaceBase.RunActionIfNoRunningPipelinesWithThreadCheck(Action action)
                    at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Boolean propagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 
                 functionsToDefine, Object[] args)
                    at System.Management.Automation.ScriptBlock.DoInvokeReturnAsIs(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Object[] args)
                    at Microsoft.PowerShell.Commands.Internal.Format.MshExpression.GetValue(PSObject target, Boolean eatExceptions)
Exception      : System.Management.Automation.RuntimeException
InvocationInfo : {}
Line           : $_.SnapshotTime.UtcDateTime.ToString("o")
Position       : At line:1 char:1
                 + $_.SnapshotTime.UtcDateTime.ToString("o")
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : -1

Message        : Exception calling "GetFullPath" with "1" argument(s): "Illegal characters in path."
StackTrace     :    at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
                    at CallSite.Target(Closure , CallSite , Type , String )
                    at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
                    at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
Exception      : System.Management.Automation.MethodInvocationException
InvocationInfo : {}
Line           : [System.IO.Path]::GetFullPath("\\?\c:\test.csv")
Position       : At line:1 char:1
                 + [System.IO.Path]::GetFullPath("\\?\c:\test.csv")
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : -1

Message        : Illegal characters in path.
StackTrace     :    at System.Security.Permissions.FileIOPermission.EmulateFileIOPermissionChecks(String fullPath)
                    at System.Security.Permissions.FileIOPermission.QuickDemand(FileIOPermissionAccess access, String fullPath, Boolean checkForDuplicates, Boolean needFullPath)
                    at CallSite.Target(Closure , CallSite , Type , String )
Exception      : System.ArgumentException
InvocationInfo : {}
Line           : [System.IO.Path]::GetFullPath("\\?\c:\test.csv")
Position       : At line:1 char:1
                 + [System.IO.Path]::GetFullPath("\\?\c:\test.csv")
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : -1

The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.
blueww commented 4 years ago

@dingmeng-xue @erich-wang Could we make the task state as "Failed" when run cmdlets with -Asjob and failure happens in EndProcessing()? We find this issue to track the ICM: https://portal.microsofticm.com/imp/v3/incidents/details/206587283/home

lymedo commented 4 years ago

I’m seeing a similar behaviour with New-AzDataLakeGen2Item. Creating a file in the root of the container but specifying / as the path results in an invalid URI error but the job state is ‘Completed’.