Issue porting page-blob to blockblob

qootec commented 2 years ago

Scenario:

Using Windows download for 0.6.20
I have a vhdx uploaded to Azure. By design, it became a page blob.
I planned to move it to archive, but page blobs can't be archived. I wasn't aware of that...
I now want to use blobporter to duplicate the blob to a blockblob and remove the pageblob afterwards (later I want to do that on several TB of VHDX that I've uploaded)

The command I tried (with some confidential info removed):

blobporter -f "https://saMyAccount.blob.core.windows.net/system20211012/Part1/Backup%202021-10-12%20073026/complexID.vhdx" -n Part1/Backup%202021-10-12%20073026/complexID.vhdx.removeExtension -c system20211012 -t blob-blockblob

Issue (with some confidential info removed):

Files to Transfer (blob-blockblob) : Source: saMyAccount.blob.core.windows.net/system20211012/Part1/Backup 2021-10-12 073026/complexID.vhdx Size:306184192 2021/11/12 16:08:13 -> github.com/Azure/azure-storage-blob-go/azblob.NewResponseError, /home/travis/gopath/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.0.0-20190123011202-457680cc0804/azblob/zz_generated_response_error.go:29 = RESPONSE ERROR (ServiceCode=InvalidBlobType) = Description=The blob type is invalid for this operation. RequestId:975a66aa-301e-015d-40d7-d72bfe000000 Time:2021-11-12T15:08:13.1203140Z, Details: Code: InvalidBlobType PUT https://saMyAccount.blob.core.windows.net/system20211012/Part1/Backup 2021-10-12 073026/complexID.vhdx?blockid=MD...someid...3D&comp=block&timeout=61 Authorization: REDACTED Content-Length: [0] User-Agent: [BlobPorter/0.6.20/amd64/Azure-SDK-For-Go/0.0.0 azblob/2018-03-28 Azure-Storage/0.5 (go1.11; Windows_NT)] X-Ms-Client-Request-Id: [ad501250-2412-481d-6b73-cdcc3ddcc887] X-Ms-Copy-Source: [saMyAccount.blob.core.windows.net/system20211012/Part1/Backup 2021-10-12 073026/complexID.vhdx?se=2021-11-12t21%3A08%3A12z&sig=REDACTED&sp=r&sr=b&sv=2018-03-28] X-Ms-Date: [Fri, 12 Nov 2021 15:08:12 GMT] X-Ms-Source-Range: [bytes=260046848-268435455] X-Ms-Version: [2018-03-28]

RESPONSE Status: 409 The blob type is invalid for this operation. Content-Length: [228] Content-Type: [application/xml] Date: [Fri, 12 Nov 2021 15:08:12 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Error-Code: [InvalidBlobType] X-Ms-Request-Id: [975a66aa-301e-015d-40d7-d72bfe000000] X-Ms-Version: [2018-03-28]

Analysis:

It feels as if the page blob is not allowed as a source, while this is the purpose of the "blob-blockblob" transfer type, I guess.
I don't see another more specific transfer type in the sources of https://github.com/Azure/blobporter/blob/master/transfer/transfer.go
I tried to specify "-n simpleName" but that doesn't help.

Any suggestions? Thanks, Johan

qootec commented 2 years ago

For your information: I've worked around the issue using "azcopy" instead of blobporter. That solution seems to work fine.

N1kolayS commented 2 years ago

Hi all! I have a same error. I tried upload large file up 45Gb blobporter -c fastq -f WGS-B-SDD130519-240402_2.fastq.gz

And i got same error

`BlobPorter 
Copyright (c) Microsoft Corporation. 
Version: 0.6.20
---------------

Files to Transfer (file-blockblob) :
Source: WGS-B-SDD130519-240402_2.fastq.gz Size:47264358266 
2021/11/18 09:10:57 -> github.com/Azure/azure-storage-blob-go/azblob.NewResponseError, /home/travis/gopath/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.0.0-20190123011202-457680cc0804/azblob/zz_generated_response_error.go:29
===== RESPONSE ERROR (ServiceCode=InvalidBlobOrBlock) =====
Description=The specified blob or block content is invalid.
RequestId:c427b903-a01e-00a3-4043-dc8f6b000000
Time:2021-11-18T06:10:57.5029439Z, Details: 
   Code: InvalidBlobOrBlock
   PUT https://wgs.blob.core.windows.net/fastq/WGS-B-SDD130519-240402_2.fastq.gz?blockid=removed_for_secure&comp=block&timeout=61
   Authorization: REDACTED
   Content-Length: [8388608]
   User-Agent: [BlobPorter/0.6.20/amd64/Azure-SDK-For-Go/0.0.0 azblob/2018-03-28 Azure-Storage/0.5 (go1.11; linux)]
   X-Ms-Client-Request-Id: [569f3fbf-5476-4703-65ee-1720f05cef9f]
   X-Ms-Date: [Thu, 18 Nov 2021 06:10:54 GMT]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 400 The specified blob or block content is invalid.
   Content-Length: [234]
   Content-Type: [application/xml]
   Date: [Thu, 18 Nov 2021 06:10:56 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [InvalidBlobOrBlock]
   X-Ms-Request-Id: [c427b903-a01e-00a3-4043-dc8f6b000000]
   X-Ms-Version: [2018-03-28]
`

forsythg commented 2 years ago

I've also had a similar issue, but in my case it's documentation related. Long story-short, it appears converting from page blobs to block blobs is only possible within the same storage account and for a whole container from one container to another:

Example - this works:

$env:ACCOUNT_NAME="<account A name>"
.\BlobPorter.exe -f "https://<account A name>.blob.core.windows.net/srccontainer" -c destcontainer -t blob-blockblob```

Example - this fails:

$env:ACCOUNT_NAME="<account B name>"
$env:SRC_ACCOUNT_KEY="abc...."
.\BlobPorter.exe -f "https://<account A name>.blob.core.windows.net/srccontainer" -c destcontainer -t blob-blockblob```

Azure / blobporter

Issue porting page-blob to blockblob #116