search

microsoft / AzureStorageExplorer

Easily manage the contents of your storage account with Azure Storage Explorer. Upload, download, and manage blobs, files, queues, tables, and Cosmos DB entities. Gain easy access to manage your virtual machine disks. Work with either Azure Resource Manager or classic storage accounts, plus manage and configure cross-origin resource sharing (CORS) rules.
Creative Commons Attribution 4.0 International
376 stars 86 forks source link

Fail to clone one append blob under one BlockBlobStorage account #2821

Open v-xianya opened 4 years ago

v-xianya commented 4 years ago

Storage Explorer Version: 1.13.0 Build: 20200410.5 Branch: rel/1.13.0 Platform/OS: Windows 10/ Linux Ubuntu 18.04/ macOS High Sierra Architecture: ia32/x64 Regression From: Previous release(1.12.0)

Steps to reproduce:

  1. Expand one BlockBlobStorage storage account -> Blob Containers.
  2. Create a blob container -> Upload one file as append blob.(make sure the file size is not 0 byte).
  3. Try to clone the uploaded append blob -> Check the result.

Expect Experience: Succeed to clone the append blob.

Actual Experience:

  1. Fail to clone the append blob.
  2. The cloned blob displays with 0 byte after manually refresh.

image Error info: This request is not authorized to perform this operation using this permission.

More Info:

  1. Fail to paste one append blob to one BlockBlobStorage accounts.
  2. Fail to clone/ paste one blob containers contains append blobs to BlockBlobStorage accounts.
MRayermannMSFT commented 4 years ago

After you make an issue in azcopy repo, move to 15 please.

JasonYeMSFT commented 4 years ago

Seems related to https://github.com/Azure/azure-storage-azcopy/issues/696

JasonYeMSFT commented 4 years ago

Move to 1.17.0 since AzCopy don't have the capacity to fix this.

jtomori commented 2 years ago

Hello, I would like to check whether there are any updates on this issue.

We are running into it when copying many files within BlockBlobStorage account with Premium performance. Some of the copied files are AppendBlobs, which cause the copy operation to fail. After the failed copy we can see empty AppendBlobs at the destination (source files aren't empty).

Attaching logs for a simplified issue reproduction: cloning a single AppendBlob.

Copy AzCopy Command to Clipboard

./azcopy.exe copy "https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?sv=2020-08-04&se=2022-02-18T13%3A49%3A43Z&sr=c&sp=rl&sig=[redacted]" "https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?sv=2020-08-04&se=2022-02-18T13%3A49%3A43Z&sr=c&sp=rwl&sig=[redacted]" --overwrite=prompt --s2s-preserve-access-tier=false --recursive --trusted-microsoft-suffixes= --log-level=INFO;
AzCopy Log File ``` 2022/01/19 13:49:43 AzcopyVersion 10.12.2 2022/01/19 13:49:43 OS-Environment windows 2022/01/19 13:49:43 OS-Architecture amd64 2022/01/19 13:49:43 Log times are in UTC. Local time is 19 Jan 2022 13:49:43 2022/01/19 13:49:43 Job-Command copy https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rl&sr=c&sv=2020-08-04 https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04 --output-type=json --cancel-from-stdin --overwrite=prompt --s2s-preserve-access-tier=false --recursive --trusted-microsoft-suffixes= --log-level=INFO 2022/01/19 13:49:43 Number of CPUs: 32 2022/01/19 13:49:43 Max file buffer RAM 16.000 GB 2022/01/19 13:49:43 Max concurrent network operations: 300 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override) 2022/01/19 13:49:43 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override) 2022/01/19 13:49:43 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override) 2022/01/19 13:49:43 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override) 2022/01/19 13:49:43 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable) 2022/01/19 13:49:43 Max open files when downloading: 2147482929 (auto-computed) 2022/01/19 13:49:43 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2022-01-19T13:49:38Z or --include-after=2022-01-19T13:49:38Z 2022/01/19 13:49:43 WARN: failed to initialize destination container [redacted]; the transfer will continue (but be wary it may fail): -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) ===== Description=This request is not authorized to perform this operation. RequestId:1db6e08d-001e-0087-463b-0d066e000000 Time:2022-01-19T13:49:43.2266643Z, Details: Code: AuthorizationFailure PUT https://[redacted].blob.core.windows.net/[redacted]?restype=container&se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04&timeout=180 User-Agent: [Microsoft Azure Storage Explorer, 1.21.3, win32, azcopy-node, 2.0.0, win32, AzCopy/10.12.2 Azure-Storage/0.14 (go1.16; Windows_NT)] X-Ms-Client-Request-Id: [d604743c-244b-419e-6f43-ce17f98fa5d5] X-Ms-Version: [2019-12-12] -------------------------------------------------------------------------------- RESPONSE Status: 403 This request is not authorized to perform this operation. Content-Length: [246] Content-Type: [application/xml] Date: [Wed, 19 Jan 2022 13:49:42 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Client-Request-Id: [d604743c-244b-419e-6f43-ce17f98fa5d5] X-Ms-Error-Code: [AuthorizationFailure] X-Ms-Request-Id: [1db6e08d-001e-0087-463b-0d066e000000] X-Ms-Version: [2019-12-12] 2022/01/19 13:49:43 Any empty folders will not be processed, because source and/or destination doesn't have full folder support 2022/01/19 13:49:43 Final job part has been created 2022/01/19 13:49:43 JobID=7393e67c-a10b-a94e-45bd-977f7757cbac, credential type: Anonymous 2022/01/19 13:49:43 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 0, E: 0, T: 0, GRs: 6 2022/01/19 13:49:43 scheduling JobID=7393e67c-a10b-a94e-45bd-977f7757cbac, Part#=0, Transfer#=0, priority=0 2022/01/19 13:49:43 Final job part has been scheduled 2022/01/19 13:49:43 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2022/01/19 13:49:43 INFO: [P#0-T#0] has worker 47 which is processing TRANSFER 0 2022/01/19 13:49:43 INFO: [P#0-T#0] Transfer: Source "https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rl&sr=c&sv=2020-08-04" Destination "https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04". Autodetected https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?se=2022-02-18T13%3A49%3A43Z&sig=-REDACTED-&sp=rl&sr=c&sv=2020-08-04 blob type as AppendBlob. 2022/01/19 13:49:43 INFO: [P#0-T#0] Starting transfer: Source "https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rl&sr=c&sv=2020-08-04" Destination "https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04". Specified chunk size 4194304 2022/01/19 13:49:43 ==> REQUEST/RESPONSE (Try=1/12.399ms, OpTime=44.3036ms) -- RESPONSE SUCCESSFULLY RECEIVED HEAD https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04&timeout=901 X-Ms-Request-Id: [42a0d1e7-801e-0042-803b-0d2c8b000000] 2022/01/19 13:49:43 ==> REQUEST/RESPONSE (Try=1/16.958ms, OpTime=16.958ms) -- RESPONSE SUCCESSFULLY RECEIVED PUT https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04&timeout=901 X-Ms-Request-Id: [42a0d1ef-801e-0042-053b-0d2c8b000000] 2022/01/19 13:49:43 ==> REQUEST/RESPONSE (Try=1/509.4µs, OpTime=509.4µs) -- RESPONSE STATUS CODE ERROR PUT https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?comp=appendblock&se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04&timeout=901 Content-Length: [0] User-Agent: [Microsoft Azure Storage Explorer, 1.21.3, win32, azcopy-node, 2.0.0, win32, AzCopy/10.12.2 Azure-Storage/0.14 (go1.16; Windows_NT)] X-Ms-Client-Request-Id: [b2fd41e1-d69a-4f66-6c8f-b81eb9fb2e02] X-Ms-Copy-Source: [https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?se=2022-02-18t13%3a49%3a43z&sig=-REDACTED-&sp=rl&sr=c&sv=2020-08-04] X-Ms-Source-Range: [bytes=0-41586] X-Ms-Version: [2020-08-04] -------------------------------------------------------------------------------- RESPONSE Status: 400 The value for one of the HTTP headers is not in the correct format. Content-Length: [322] Content-Type: [application/xml] Date: [Wed, 19 Jan 2022 13:49:42 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Client-Request-Id: [b2fd41e1-d69a-4f66-6c8f-b81eb9fb2e02] X-Ms-Error-Code: [InvalidHeaderValue] X-Ms-Request-Id: [42a0d1fe-801e-0042-0e3b-0d2c8b000000] X-Ms-Version: [2020-08-04] Response Details: InvalidHeaderValueThe value for one of the HTTP headers is not in the correct format. Content-Length0 2022/01/19 13:49:43 ERR: [P#0-T#0] COPYFAILED: https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rl&sr=c&sv=2020-08-04 : 400 : 400 The value for one of the HTTP headers is not in the correct format.. When Appending block from URL. X-Ms-Request-Id: 42a0d1fe-801e-0042-0e3b-0d2c8b000000 Dst: https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04 2022/01/19 13:49:43 ==> REQUEST/RESPONSE (Try=1/505.4µs, OpTime=505.4µs) -- RESPONSE STATUS CODE ERROR DELETE https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04&timeout=31 User-Agent: [Microsoft Azure Storage Explorer, 1.21.3, win32, azcopy-node, 2.0.0, win32, AzCopy/10.12.2 Azure-Storage/0.14 (go1.16; Windows_NT)] X-Ms-Client-Request-Id: [6432e885-ba20-460f-48c5-11aeb3c9c44d] X-Ms-Version: [2020-08-04] -------------------------------------------------------------------------------- RESPONSE Status: 403 This request is not authorized to perform this operation using this permission. Content-Length: [279] Content-Type: [application/xml] Date: [Wed, 19 Jan 2022 13:49:42 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Client-Request-Id: [6432e885-ba20-460f-48c5-11aeb3c9c44d] X-Ms-Error-Code: [AuthorizationPermissionMismatch] X-Ms-Request-Id: [42a0d204-801e-0042-123b-0d2c8b000000] X-Ms-Version: [2020-08-04] Response Details: AuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission. 2022/01/19 13:49:43 ERR: [P#0-T#0] https://[redacted].blob.core.windows.net/[redacted]/[redacted]/stdout.txt_copy_4?se=2022-02-18t13%3A49%3A43z&sig=-REDACTED-&sp=rwl&sr=c&sv=2020-08-04: 403: Delete (incomplete) Append Blob -403 This request is not authorized to perform this operation using this permission.. X-Ms-Request-Id:42a0d204-801e-0042-123b-0d2c8b000000 2022/01/19 13:49:43 JobID=7393e67c-a10b-a94e-45bd-977f7757cbac, Part#=0, TransfersDone=1 of 1 2022/01/19 13:49:43 all parts of entire Job 7393e67c-a10b-a94e-45bd-977f7757cbac successfully completed, cancelled or paused 2022/01/19 13:49:43 is part of Job which 1 total number of parts done 2022/01/19 13:49:45 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 0, E: 0, T: 0, GRs: 300 ```

I can copy/clone an AppendBlob on a StorageV2 account with Standard/Cool performance/tier without any issues.

Using ASE 1.21.3 with azcopy 10.12.2 on Windows.

jtomori commented 2 years ago

Can you think of any temporary fix for this?

In one of our scenarios we don't care about AppendBlobs, so excluding them, or silencing copy errors would help us too. This would let us check ASE's transfer status for any potential issues, e.g. BlockBlobs we care about.

Azcopy has the --exclude-blob-type=AppendBlob flag which looks like what we need, but I haven't found a way to instruct ASE to add that flag when internally calling azcopy.

Azcopy called by ASE seems to respect environment variables, but I haven't found a variable for excluding AppendBlobs.

I've also found that ASE supports some sort of extensions, which could potentially help in this case, but haven't found any documentation for that.

Another alternative is to build some custom gui around azcopy, but that would lack a lot of ASE's functionality. Note that our users aren't programmers or too comfortable with command line.

JasonYeMSFT commented 2 years ago

Unfortunately the most straightforward way to mitigate this problem is to directly use the command line tool for AzCopy. Sorting by blob type might help you manually exclude append blobs but that would be very tedious if you have a large number of blobs.

jtomori commented 2 years ago

I see, thank you for your reply.

One more observation: copying AppendBlob with a cp command on a linux machine with a Premium BlockBlobStorage container mounted with blobfuse succeeds, but the resulting blob is of BlockBlob type.