search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.26k stars 3.86k forks source link

[Az.Storage] Move-AzDataLakeGen2Item fails with 401 (Server failed to authenticate the request.) while using SAS token #14875

Open blueww opened 3 years ago

blueww commented 3 years ago

Description

Move-AzDataLakeGen2Item fails with 403 while using SAS token. This is because we also need input SAS token for dest item, but SDK don't provide API to input the dest SAS token.

Steps to reproduce

     # Prepare: Create SAS from account key 
      $ctx = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $key 
      $sas = New-AzStorageContainerSASToken -Name $filesystemName -Permission rwdl -Context $ctx
      $sasctx = New-AzStorageContext -StorageAccountName $ctx.StorageAccountName -SasToken $sas

       $dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $filesystemName -Path $dirname1 -DestFileSystem $filesystemName -DestPath $dirname3

Environment data

$PSVersionTable won't impact

Module versions

Az.Storage 3.6.0

Debug output

The error output has enough information

Error output

        $dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $filesystemName -Path $dirname1 -DestFileSystem $filesystemName -DestPath $dir3
Move-AzDataLakeGen2Item : Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:756e9c33-d01f-0077-1e68-3d9834000000
Time:2021-04-30T02:25:46.8840310Z
Status: 401 (Server failed to authenticate the request. Please refer to the information in the www-authenticate header.)
ErrorCode: NoAuthenticationInformation
Headers:
x-ms-error-code: NoAuthenticationInformation
x-ms-request-id: 756e9c33-d01f-0077-1e68-3d9834000000
x-ms-version: 2020-04-08
x-ms-client-request-id: a4dbb40b-0338-45e4-851d-17e5b72bfb40
Content-Length: 250
Content-Type: application/json;charset=utf-8
Date: Fri, 30 Apr 2021 02:25:46 GMT
Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0
WWW-Authenticate: REDACTED
At line:2 char:17
+ ...     $dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $fil ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Move-AzDataLakeGen2Item], RequestFailedException
    + FullyQualifiedErrorId : RequestFailedException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.MoveAzDataLakeGen2ItemCommand
blueww commented 3 years ago

The workaround is input the dest sas together with destpath in like -DestPath "$($dirname3)$($sas)" ，see following sample: Note: There’s a limitation in PSH: when you input -Destpath in this way, Powershell will always think the dest path not exist, so won’t ask user to confirm overwrite even the dest path exist. (Powershell will take destpath+sastoken as path name in check dest path exist.)

# Prepare: Create SAS from account key 
$ctx = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $key 
$sas = New-AzStorageContainerSASToken -Name $filesystemName -Permission rwdl -Context $ctx
$sasctx = New-AzStorageContext -StorageAccountName $ctx.StorageAccountName -SasToken $sas

# this sas token created with New-AzStorageContainerSASToken has prefix “?”. If sas token doesn’t have prefix “?”, need use -DestPath "$($dirname3)?$($sas)"
$dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $filesystemName -Path $dirname1 -DestFileSystem $filesystemName -DestPath "$($dirname3)$($sas)"

We will consider to give a more formal fix after SDK fix this. After the fix, the above workaround might not work anymore.

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams.

Issue Details
## Description Move-AzDataLakeGen2Item fails with 403 while using SAS token. This is because we also need input SAS token for dest item, but SDK don't provide API to input the dest SAS token. ## Steps to reproduce ```powershell # Prepare: Create SAS from account key $ctx = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $key $sas = New-AzStorageContainerSASToken -Name $filesystemName -Permission rwdl -Context $ctx $sasctx = New-AzStorageContext -StorageAccountName $ctx.StorageAccountName -SasToken $sas $dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $filesystemName -Path $dirname1 -DestFileSystem $filesystemName -DestPath $dirname3 ``` ## Environment data ``` $PSVersionTable won't impact ``` ## Module versions ```powershell Az.Storage 3.6.0 and before ``` ## Debug output ``` ``` ## Error output ``` $dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $filesystemName -Path $dirname1 -DestFileSystem $filesystemName -DestPath $dir3 Move-AzDataLakeGen2Item : Server failed to authenticate the request. Please refer to the information in the www-authenticate header. RequestId:756e9c33-d01f-0077-1e68-3d9834000000 Time:2021-04-30T02:25:46.8840310Z Status: 401 (Server failed to authenticate the request. Please refer to the information in the www-authenticate header.) ErrorCode: NoAuthenticationInformation Headers: x-ms-error-code: NoAuthenticationInformation x-ms-request-id: 756e9c33-d01f-0077-1e68-3d9834000000 x-ms-version: 2020-04-08 x-ms-client-request-id: a4dbb40b-0338-45e4-851d-17e5b72bfb40 Content-Length: 250 Content-Type: application/json;charset=utf-8 Date: Fri, 30 Apr 2021 02:25:46 GMT Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0 WWW-Authenticate: REDACTED At line:2 char:17 + ... $dir3 = Move-AzDataLakeGen2Item -Context $sasctx -FileSystem $fil ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Move-AzDataLakeGen2Item], RequestFailedException + FullyQualifiedErrorId : RequestFailedException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.MoveAzDataLakeGen2ItemCommand ```
Author: blueww
Assignees: -
Labels: `Data Lake Storage Gen2`, `Service Attention`, `needs-triage`
Milestone: -
hkelley commented 3 years ago

I have a slightly different (403 and "Signature did not match. String to sign used was ...") error when I am using New-AzStorageContainerSASToken. In my case, the returned token already has the "?" prefix, but I still get this exception.

New-AzStorageContainerSASToken -Context $signingContext -Name $containerName -Permission rcwl -Protocol HttpsOnly -StartTime (Get-Date) -ExpiryTime (Get-Date).AddDays(70)

results in this token:

?sv=2019-07-07&sr=c&sig=XXXXXXXXXXXX%2BtpBQaDPrfPU%3D&spr=https&st=2021-07-19T13%3A28%3A34Z&se=2021-09-27T13%3A28%3A34Z&sp=rcwl

An AzStorageContext generated from that SAS token appears to be similar to a context generated from a UI-calculated token, but the cmdlets throw this exception when I use the context:

Set-AzStorageBlobContent : Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. HTTP Status Code: 403 - HTTP Error Message: Server failed to authenticate the request. Make sure 
the value of Authorization header is formed correctly including the signature.
ErrorCode: AuthenticationFailed
ErrorMessage: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Time:2021-07-19T13:28:58.5864077Z
AuthenticationErrorDetail: Signature did not match. String to sign used was rcwl
2021-07-19T13:28:34Z
2021-09-27T13:28:34Z
.....
blueww commented 3 years ago

@hkelley I can't repro this issue with latest Az.Storage 3.9.0.

Following is my repro script:

$ctx = New-AzStorageContext -StorageAccountName $accountname -StorageAccountKey $accountkey

$sas = New-AzStorageContainerSASToken -Context $ctx -Name $containerName -Permission rcwl -Protocol HttpsOnly -StartTime (Get-Date) -ExpiryTime (Get-Date).AddDays(70)
$sasctx  = New-AzStorageContext -StorageAccountName $accountname -SasToken $sas

Set-AzStorageBlobContent -Container $containerName -Blob testblob -File $localSrcFile -Context $sasctx -Force

Would you please share:

  1. The Az.Storage module version (Get-module)
  2. The Powershell version ($PSVersionTable)
  3. It seems the error message from you missing request ID. Would you please add the RequestId (the error message should has it). So I can look at server log to see if any hint.
  4. More detail script:
    • How do you create "$signingContext", is it with key credential?
    • How do you create the sas storage context
    • the detail command to run Set-AzStorageBlobContent

As this is for a different issue than the original one, would you please open a new issue, and if you following the issue template, we should can get most information needed for investigation.