search

pivotal-cf / azure-blobstore-resource

A concourse resource to interact with the azure blob service.
MIT License
16 stars 15 forks source link

Parallel Uploads - The specified block list is invalid #37

Closed dawu415 closed 3 years ago

dawu415 commented 3 years ago

Running multiple uploads from multiple jobs running in parallel sometimes gives the following error. We did set the block_size of 100Mb. Not sure if it is related to this StackOverflow - The specified block list is invalid )

2020/11/05 14:58:06 failed to upload blob: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=InvalidBlockList) =====
Description=The specified block list is invalid.
RequestId:8fdbe99d-d01e-0004-2484-b3c323000000
Time:2020-11-05T14:58:06.4775739Z, Details: 
   Code: InvalidBlockList
   PUT https://xxxxx.blob.core.windows.net/product-tiles/stemcells/[stemcells-ubuntu-xenial,621.90]bosh-stemcell-621.90-azure-hyperv-ubuntu-xenial-go_agent.tgz?comp=blocklist&timeout=61
   Authorization: REDACTED
   Content-Length: [653]
   Content-Type: [application/xml]
   User-Agent: [Azure-Storage/0.7 (go1.14.2; linux)]
   X-Ms-Blob-Cache-Control: []
   X-Ms-Blob-Content-Disposition: []
   X-Ms-Blob-Content-Encoding: []
   X-Ms-Blob-Content-Language: []
   X-Ms-Blob-Content-Type: []
   X-Ms-Client-Request-Id: [599d2426-0cbd-4acb-60e5-2d8db6fecfb2]
   X-Ms-Date: [Thu, 05 Nov 2020 14:58:06 GMT]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 400 The specified block list is invalid.
   Content-Length: [221]
   Content-Type: [application/xml]
   Date: [Thu, 05 Nov 2020 14:58:05 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [InvalidBlockList]
   X-Ms-Request-Id: [8fdbe99d-d01e-0004-2484-b3c323000000]
   X-Ms-Version: [2018-11-09]
christianang commented 3 years ago

That is an interesting problem. Unfortunately, I don't have the bandwidth to work on this, so if you would like to make a PR to fix this, that I can validate works, then I can merge and make a new release.

lancefrench commented 3 years ago

I think this may have been fixed in azure-storage-blob-go@v0.10.0. @dawu415, I'd be happy to work with you on building a new version of the resource for testing and submit a pull request if either your bug or streaming performance are improved by bumping to 0.10.0 or 0.11.0.

christianang commented 3 years ago

I've created a new release (v0.11.0) that has the changes made by @lancefrench in #38.