Export/Import-AzDataLakeStoreItem hangs when source doesn't exist

Azure / azure-powershell

Microsoft Azure PowerShell

Other

4.26k stars 3.86k forks source link

Description

When using powershell+ADL in ubuntu, Export-AzDataLakeStoreItem will hang if source file or folder doesn't exists. It happens when:

using script pwsh -f script.file
or using inline command pwsh -Command "Export-AzDataLakeStoreItem ..."

Screen printed error message saying file doesn't exist, but after it, program hangs there forever. Even ctr+c can't kill it. I used ctr+z to put it in the background, then kill it by "kill %n".

Hovever, when using interactive shell mode, no such problem.

Same for operation "Import-AzDataLakeStoreItem".

Steps to reproduce

pwsh -Command "Export-AzDataLakeStoreItem -AccountName bingads-data-algo-prod-c08 -Destination ./temp -P
ath /local/non-exist-file-or-folder/ -Debug"

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.1.0
PSEdition                      Core
GitCommitId                    7.1.0
OS                             Linux 5.4.0-1031-azure #32~18.04.1-Ubuntu SMP Tue Oct 6 10:03:22 UTC 2020
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

PowerShell 7.1.0

> Get-Module  -ListAvailable

    Directory: /root/.local/share/powershell/Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     2.2.1                 Az.Accounts                         Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollecti…
Script     1.3.0                 Az.DataLakeStore                    Core,Desk {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvi…

    Directory: /opt/microsoft/powershell/7/Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Manifest   1.2.5                 Microsoft.PowerShell.Archive        Desk      {Compress-Archive, Expand-Archive}
Manifest   7.0.0.0               Microsoft.PowerShell.Host           Core      {Start-Transcript, Stop-Transcript}
Manifest   7.0.0.0               Microsoft.PowerShell.Management     Core      {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path…}
Manifest   7.0.0.0               Microsoft.PowerShell.Security       Core      {Get-Credential, Get-ExecutionPolicy, Set-ExecutionPolicy, ConvertFrom-Secu…
Manifest   7.0.0.0               Microsoft.PowerShell.Utility        Core      {Export-Alias, Get-Alias, Import-Alias, New-Alias…}
Script     1.4.7                 PackageManagement                   Desk      {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource…}
Script     2.2.5                 PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-Module, Find-RoleCapability…}
Script     2.0.5                 PSDesiredStateConfiguration         Core      {Configuration, New-DscChecksum, Get-DscResource, Invoke-DscResource}
Script     2.1.0                 PSReadLine                          Desk      {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHa…
Binary     2.0.3                 ThreadJob                           Desk      Start-ThreadJob

Debug output

# pwsh -Command "Export-AzDataLakeStoreItem -AccountName bingads-data-algo-prod-c08 -Destination ./temp -P
ath /local/non-exist-file-or-folder/ -Debug"
DEBUG: Sought all Az modules and got latest version 0.0.0
DEBUG: 17:57:43 - ExportAzureDataLakeStoreItem begin processing with ParameterSet 'NoDiagnosticLogging'.
DEBUG: 17:57:43 - using account id '24afef91-67f0-450c-8f3f-2b6d8e81bf27'...
WARNING: Upcoming breaking changes in the cmdlet 'Export-AzDataLakeStoreItem' :
For store side export failures, Export-AzDataLakeStoreItem will throw exception instead of printing message on screen
- This change will take effect on '05/18/2020 00:00:00'
- The change is expected to take effect from the version : '2.0.0'
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

Confirm
Continue with this operation?
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"):

Confirm
Are you sure you want to perform this action?
Performing the operation "Export" on target "/local/non-exist-file-or-folder".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
DEBUG: [Common.Authentication]: Authenticating using Account: '24afef91-67f0-450c-8f3f-2b6d8e81bf27', environment: 'AzureCloud', tenant: '72f988bf-86f1-41af-91ab-2d7cd011db47'
DEBUG: ClientSecretCredential.GetToken invoked. Scopes: [ https://datalake.azure.net//.default ] ParentRequestId:
DEBUG: Request [2bf511af-1961-4d75-a1e0-5e402655b61d] GET https://login.microsoftonline.com/common/discovery/instance?api-version=REDACTED&authorization_endpoint=REDACTED
x-client-SKU:REDACTED
x-client-Ver:REDACTED
x-client-OS:REDACTED
client-request-id:REDACTED
return-client-request-id:REDACTED
x-app-name:REDACTED
x-app-ver:REDACTED
x-ms-client-request-id:2bf511af-1961-4d75-a1e0-5e402655b61d
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.4.0-beta.1,(.NET 5.0.0; Linux 5.4.0-1031-azure #32~18.04.1-Ubuntu SMP Tue Oct 6 10:03:22 UTC 2020)
client assembly: Azure.Identity
DEBUG: Response [2bf511af-1961-4d75-a1e0-5e402655b61d] 200 OK (00.3s)
Cache-Control:max-age=86400, private
Strict-Transport-Security:REDACTED
X-Content-Type-Options:REDACTED
Access-Control-Allow-Origin:REDACTED
Access-Control-Allow-Methods:REDACTED
P3P:REDACTED
client-request-id:REDACTED
x-ms-request-id:REDACTED
x-ms-ests-server:REDACTED
Set-Cookie:REDACTED
Date:Tue, 01 Dec 2020 17:58:04 GMT
Content-Type:application/json; charset=utf-8
Content-Length:980

DEBUG: Request [06377b1d-15e0-4d58-a71e-5aaa4e348223] POST https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token
x-client-SKU:REDACTED
x-client-Ver:REDACTED
x-client-OS:REDACTED
x-client-current-telemetry:REDACTED
x-client-last-telemetry:REDACTED
x-ms-PKeyAuth:REDACTED
x-ms-lib-capability:REDACTED
client-request-id:REDACTED
return-client-request-id:REDACTED
x-app-name:REDACTED
x-app-ver:REDACTED
x-ms-client-request-id:06377b1d-15e0-4d58-a71e-5aaa4e348223
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.4.0-beta.1,(.NET 5.0.0; Linux 5.4.0-1031-azure #32~18.04.1-Ubuntu SMP Tue Oct 6 10:03:22 UTC 2020)
client assembly: Azure.Identity
DEBUG: Response [06377b1d-15e0-4d58-a71e-5aaa4e348223] 200 OK (00.2s)
Cache-Control:no-store, no-cache
Pragma:no-cache
Strict-Transport-Security:REDACTED
X-Content-Type-Options:REDACTED
P3P:REDACTED
client-request-id:REDACTED
x-ms-request-id:REDACTED
x-ms-ests-server:REDACTED
x-ms-clitelem:REDACTED
Set-Cookie:REDACTED
Date:Tue, 01 Dec 2020 17:58:04 GMT
Content-Type:application/json; charset=utf-8
Expires:-1
Content-Length:11234

DEBUG: ClientSecretCredential.GetToken succeeded. Scopes: [ https://datalake.azure.net//.default ] ParentRequestId:  ExpiresOn: 2020-12-02T17:58:03.0000000+00:00
DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: '72f988bf-86f1-41af-91ab-2d7cd011db47', UserId: '24afef91-67f0-450c-8f3f-2b6d8e81bf27'
DEBUG: 2020-12-01 17:58:04.6998|DEBUG|adls.dotnet.FileTransfer|FileTransfer.Downloader, Src: /local/non-exist-file-or-folder, Dest: /datadrive/jiuchen/src/to.be.deleted/shc/temp, Threads: 128, TrackingProgress: True, OverwriteIfExist: False
DEBUG: 2020-12-01 17:58:04.8887|DEBUG|adls.powershell.WebTransport|VERB: GET

RequestHeaders:
[AuthorizationHeaderLength:11139]
[User-Agent:Microsoft.Azure.DataLake.Store;1.2.4-alpha/Linux 5.4.0-1031-azure #32~18.04.1-Ubuntu SMP Tue Oct 6 10:03:22 UTC 2020 X64/NETSTANDARD1_4/AzurePowershell/Az1.0.0]
[x-ms-client-request-id:51735d6e-abe3-488b-a743-5cdb2f69261f.0]

ResponseStatus:NotFound

ResponseHeaders:
[Cache-Control:no-store, no-cache, max-age=0]
[Pragma:no-cache]
[x-ms-request-id:ac7f1370-a112-46af-91f7-99372523bcb3]
[x-ms-webhdfs-version:17.04.24.00]
[Status:0x8309000A]
[X-Content-Type-Options:nosniff]
[Strict-Transport-Security:max-age=15724800; includeSubDomains]
[Date:Tue, 01 Dec 2020 17:58:04 GMT]

DEBUG: 2020-12-01 17:58:04.8914|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:51735d6e-abe3-488b-a743-5cdb2f69261f.0,lat:177,err:HTTP NotFound ( FileNotFoundException  File/Folder does not exist: /local/non-exist-file-or-folder [ac7f1370-a112-46af-91f7-99372523bcb3][2020-12-01T09:58:04.8711379-08:00] JavaClassName: java.io.FileNotFoundException ,Reqlen:0,Resplen:0,token_ns:2,sReqId:ac7f1370-a112-46af-91f7-99372523bcb3,path:/local/non-exist-file-or-folder,qp:op=GETFILESTATUS&tooid=True&api-version=2018-09-01, connectionfailure:False
Export-AzDataLakeStoreItem: ADLSException: Error in getting metadata for path /local/non-exist-file-or-folder.
Operation: GETFILESTATUS failed with HttpStatus:NotFound  RemoteException: FileNotFoundException File/Folder does not exist: /local/non-exist-file-or-folder [ac7f1370-a112-46af-91f7-99372523bcb3][2020-12-01T09:58:04.8711379-08:00] JavaClassName: java.io.FileNotFoundException.
Last encountered exception thrown after 1 tries. [HTTP NotFound ( FileNotFoundException  File/Folder does not exist: /local/non-exist-file-or-folder [ac7f1370-a112-46af-91f7-99372523bcb3][2020-12-01T09:58:04.8711379-08:00] JavaClassName: java.io.FileNotFoundException ]
[ServerRequestId:ac7f1370-a112-46af-91f7-99372523bcb3]
DEBUG: AzureQoSEvent: CommandName - Export-AzDataLakeStoreItem; IsSuccess - False; Duration - 00:00:21.1046699;; Exception - Microsoft.Rest.Azure.CloudException: ADLSException: Error in getting metadata for path /local/non-exist-file-or-folder.
Operation: GETFILESTATUS failed with HttpStatus:NotFound  RemoteException: FileNotFoundException File/Folder does not exist: /local/non-exist-file-or-folder [ac7f1370-a112-46af-91f7-99372523bcb3][2020-12-01T09:58:04.8711379-08:00] JavaClassName: java.io.FileNotFoundException.
Last encountered exception thrown after 1 tries. [HTTP NotFound ( FileNotFoundException  File/Folder does not exist: /local/non-exist-file-or-folder [ac7f1370-a112-46af-91f7-99372523bcb3][2020-12-01T09:58:04.8711379-08:00] JavaClassName: java.io.FileNotFoundException ]
[ServerRequestId:ac7f1370-a112-46af-91f7-99372523bcb3]
   at Microsoft.Azure.Commands.DataLakeStore.ExportAzureDataLakeStoreItem.<>c__DisplayClass38_0.<ExecuteCmdlet>b__0()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(String processMessage, String target, Action action)
   at Microsoft.Azure.Commands.DataLakeStore.ExportAzureDataLakeStoreItem.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 17:58:06 - ExportAzureDataLakeStoreItem end processing.

Error output

See above.

Author:	JiushengChen
Assignees:	dingmeng-xue
Labels:	`Data Lake Store`, `Service Attention`, `needs-triage`
Milestone:	-

Author:

JiushengChen

Assignees:

dingmeng-xue

Labels:

`Data Lake Store`, `Service Attention`, `needs-triage`

Milestone:

Azure / azure-powershell