search

Azure / blobporter

Highly concurrent data transfer tool for Azure Blob Storage.
Other
85 stars 29 forks source link

Block Size is too small #71

Closed jkergosi closed 6 years ago

jkergosi commented 6 years ago

I'm using blobporter to move a 300 GB file to Azure. The command works on other files, but when I try the large one, it gives the error:

Block size is too small, minimum block size for this file would be 6442737

I added the attribute -b 100MB, but it's still giving the error. Also, I tried few various block size but no luck.

Ex: BlobPorter.exe -f MYBACKUP.FUL -t file-pageblob -b 100MB -a MYACCOUNT -c MYCONTAINER -k MYKEY

NOTE: this is a page blob, so I'm not even sure why blob blocks are being considered.

But if it mattered for page blobs, there's still a problem:

The documentation says "The maximum block size is 100MB" If I presume "min of 6442737" means Bytes, then that is 6,442,737 B = 6.4MB, well below the 100MB threshold. Even if it were over, that would be a hindrance to have a max file size allowed for Azure blob upload.

Any ideas?

Thanks, Jason

giventocode commented 6 years ago

The maximum size block size for a page blob is 4MB. When you set the value higher than that, blobporter adjusts the block size to 4MB. However, you are hitting a check that should not apply for a page blob, as it is intended for block blobs, due to their limit of 50000 blocks, which at 4MB results in a maximum blob size of ~200GB. This is an issue. We will fix this issue in the next minor release. Thanks for reporting it!

giventocode commented 6 years ago

This should be fixed in 0.5.14.