Closed hashim-hivery closed 2 weeks ago
My code, as it stands, does work with the real azure blob storage, just doesn't work with azurite
@hashim-hivery
I can't repro this issue with latest Azurite. In my test, the block ID will always be added to the StringTosign.
And per Azurite code here, you can see the Uri Query parameters will be added to StringTosign.
Azurite debug log in my test, you can see StringToSign contains blockid.
2024-03-14T03:54:49.285Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/weitest/test.txt?comp=block&blockid=M1JpL09vNEtLajVUNG1ZY2Y3VTFOQT09MTIzNDU2NzgtMDAwMDIz&timeout=300 RequestHeaders:{"x-ms-client-request-id":"Azure-Storage-PowerShell-29e2590e-7d41-4f7b-88e2-51262206f2c4","user-agent":"AzurePowershell/v1.0.0 DataMovement/2.0.0.3 Azure-Storage/11.2.2 (.NET Core; Win32NT 10.0.22631.0)","x-ms-version":"2019-07-07","x-ms-date":"Thu, 14 Mar 2024 03:54:49 GMT","authorization":"SharedKey devstoreaccount1:QAREC9rk0mXO0WRuzfBDN24ghRwy088rA0HqGKxBgeY=","content-md5":"UeMgOhJLUI4E2FskZHFWcw==","host":"127.0.0.1:10000","content-length":"8388608"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2024-03-14T03:54:49.287Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=weitest Blob=test.txt
2024-03-14T03:54:49.287Z f64ff023-4e23-47a1-9452-9eca36e93d9a verbose: DispatchMiddleware: Dispatching request...
2024-03-14T03:54:49.289Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: DispatchMiddleware: Operation=BlockBlob_StageBlock
2024-03-14T03:54:49.290Z f64ff023-4e23-47a1-9452-9eca36e93d9a verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2024-03-14T03:54:49.291Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: PublicAccessAuthenticator:validate() Start validation against public access.
2024-03-14T03:54:49.291Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: PublicAccessAuthenticator:validate() Getting account properties...
2024-03-14T03:54:49.292Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: weitest, blob: test.txt
2024-03-14T03:54:49.293Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container weitest
2024-03-14T03:54:49.294Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2024-03-14T03:54:49.294Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n8388608\nUeMgOhJLUI4E2FskZHFWcw==\n\n\n\n\n\n\n\nx-ms-client-request-id:Azure-Storage-PowerShell-29e2590e-7d41-4f7b-88e2-51262206f2c4\nx-ms-date:Thu, 14 Mar 2024 03:54:49 GMT\nx-ms-version:2019-07-07\n/devstoreaccount1/devstoreaccount1/weitest/test.txt\nblockid:M1JpL09vNEtLajVUNG1ZY2Y3VTFOQT09MTIzNDU2NzgtMDAwMDIz\ncomp:block\ntimeout:300"
2024-03-14T03:54:49.295Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:QAREC9rk0mXO0WRuzfBDN24ghRwy088rA0HqGKxBgeY=
2024-03-14T03:54:49.295Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2024-03-14T03:54:49.296Z f64ff023-4e23-47a1-9452-9eca36e93d9a verbose: DeserializerMiddleware: Start deserializing...
2024-03-14T03:54:49.297Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: HandlerMiddleware: DeserializedParameters={"blockId":"M1JpL09vNEtLajVUNG1ZY2Y3VTFOQT09MTIzNDU2NzgtMDAwMDIz","options":{"timeout":300,"transactionalContentMD5":{"type":"Buffer","data":[81,227,32,58,18,75,80,142,4,216,91,36,100,113,86,115]},"requestId":"Azure-Storage-PowerShell-29e2590e-7d41-4f7b-88e2-51262206f2c4","leaseAccessConditions":{},"cpkInfo":{},"cpkScopeInfo":{}},"comp":"block","contentLength":8388608,"version":"2019-07-07","body":"ReadableStream"}
2024-03-14T03:54:49.297Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: OperationQueue.operate() Schedule incoming job e0bb303d-e61a-4eba-a520-765d3db0e68b
2024-03-14T03:54:49.297Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: OperationQueue:execute() Current runningConcurrency:3 maxConcurrency:50 operations.length:1
2024-03-14T03:54:49.298Z f64ff023-4e23-47a1-9452-9eca36e93d9a info: FSExtentStore:appendExtent() Select extent from idle location for extent append operation. LocationId:4 extentId:95ef8be7-c365-4d8a-adae-983adfb9d13a offset:16777216 MAX_EXTENT_SIZE:67108864
2024-03-14T03:54:49.298Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: FSExtentStore:appendExtent() Get fd:7 for extent:95ef8be7-c365-4d8a-adae-983adfb9d13a from cache.
2024-03-14T03:54:49.298Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: FSExtentStore:appendExtent() Created write stream for fd:7
2024-03-14T03:54:49.299Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: FSExtentStore:appendExtent() Start writing to extent 95ef8be7-c365-4d8a-adae-983adfb9d13a
2024-03-14T03:54:49.299Z f64ff023-4e23-47a1-9452-9eca36e93d9a debug: FSExtentStore:streamPipe() Start piping data to write stream
@blueww Maybe there's some parsing problem somewhere, can you try with the same block id that I used?
Hi @hashim-hivery ,
I tried with the same block id you provided, it also works well.
2024-03-15T09:22:25.657Z 0aeecaef-052f-4754-a894-752c6a986a05 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/container171049454435304424/blob171049454440503068?comp=block&blockid=MDAwMDE%3D RequestHeaders:{"x-ms-version":"2023-08-03","content-length":"10","content-type":"application/octet-stream","accept":"application/xml","user-agent":"azsdk-js-storageblob/12.16.0 (NODE-VERSION v18.19.0; Windows_NT 10.0.22631)","x-ms-client-request-id":"e4147f27-41db-46ae-9312-a4a5deb3206f","x-ms-date":"Fri, 15 Mar 2024 09:22:25 GMT","authorization":"SharedKey devstoreaccount1:pFuvy+348iEQKLNgMbWZuRbL0v9Ug9BevGEgNIuNEM4=","host":"127.0.0.1:11000","connection":"close"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2024-03-15T09:22:25.658Z 0aeecaef-052f-4754-a894-752c6a986a05 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=container171049454435304424 Blob=blob171049454440503068
2024-03-15T09:22:25.659Z 0aeecaef-052f-4754-a894-752c6a986a05 verbose: DispatchMiddleware: Dispatching request...
2024-03-15T09:22:25.662Z 0aeecaef-052f-4754-a894-752c6a986a05 info: DispatchMiddleware: Operation=BlockBlob_StageBlock
2024-03-15T09:22:25.664Z 0aeecaef-052f-4754-a894-752c6a986a05 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2024-03-15T09:22:25.665Z 0aeecaef-052f-4754-a894-752c6a986a05 info: PublicAccessAuthenticator:validate() Start validation against public access.
2024-03-15T09:22:25.665Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: PublicAccessAuthenticator:validate() Getting account properties...
2024-03-15T09:22:25.665Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: container171049454435304424, blob: blob171049454440503068
2024-03-15T09:22:25.669Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container container171049454435304424
2024-03-15T09:22:25.670Z 0aeecaef-052f-4754-a894-752c6a986a05 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2024-03-15T09:22:25.671Z 0aeecaef-052f-4754-a894-752c6a986a05 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n10\n\napplication/octet-stream\n\n\n\n\n\n\nx-ms-client-request-id:e4147f27-41db-46ae-9312-a4a5deb3206f\nx-ms-date:Fri, 15 Mar 2024 09:22:25 GMT\nx-ms-version:2023-08-03\n/devstoreaccount1/devstoreaccount1/container171049454435304424/blob171049454440503068\nblockid:MDAwMDE=\ncomp:block"
2024-03-15T09:22:25.671Z 0aeecaef-052f-4754-a894-752c6a986a05 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:pFuvy+348iEQKLNgMbWZuRbL0v9Ug9BevGEgNIuNEM4=
2024-03-15T09:22:25.672Z 0aeecaef-052f-4754-a894-752c6a986a05 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2024-03-15T09:22:25.672Z 0aeecaef-052f-4754-a894-752c6a986a05 verbose: DeserializerMiddleware: Start deserializing...
2024-03-15T09:22:25.673Z 0aeecaef-052f-4754-a894-752c6a986a05 info: HandlerMiddleware: DeserializedParameters={"blockId":"MDAwMDE=","options":{"requestId":"e4147f27-41db-46ae-9312-a4a5deb3206f","leaseAccessConditions":{},"cpkInfo":{},"cpkScopeInfo":{}},"comp":"block","contentLength":10,"version":"2023-08-03","body":"ReadableStream"}
2024-03-15T09:22:25.675Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue.operate() Schedule incoming job ff19fdb8-252b-4eee-aa64-edcb42ec4421
2024-03-15T09:22:25.676Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:10 operations.length:1
2024-03-15T09:22:25.677Z 0aeecaef-052f-4754-a894-752c6a986a05 info: FSExtentStore:appendExtent() Select extent from idle location for extent append operation. LocationId:1 extentId:342da729-4d82-4c3a-b7bd-0d8a20cfc073 offset:0 MAX_EXTENT_SIZE:67108864
2024-03-15T09:22:25.677Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:appendExtent() Get fd:undefined for extent:342da729-4d82-4c3a-b7bd-0d8a20cfc073 from cache.
2024-03-15T09:22:25.679Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:appendExtent() Open file:__test_blob_extent__\342da729-4d82-4c3a-b7bd-0d8a20cfc073 for extent:342da729-4d82-4c3a-b7bd-0d8a20cfc073, get new fd:3
2024-03-15T09:22:25.681Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:appendExtent() Created write stream for fd:3
2024-03-15T09:22:25.681Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:appendExtent() Start writing to extent 342da729-4d82-4c3a-b7bd-0d8a20cfc073
2024-03-15T09:22:25.682Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:streamPipe() Start piping data to write stream
2024-03-15T09:22:25.683Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:streamPipe() Readable stream triggers close event, 10 bytes piped
2024-03-15T09:22:25.684Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:streamPipe() Invoke write stream end()
2024-03-15T09:22:25.684Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:streamPipe() Readable stream triggers close event, 10 bytes piped
2024-03-15T09:22:25.685Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:streamPipe() Writable stream triggers finish event, after 10 bytes piped. Flush data to fd:3.
2024-03-15T09:22:25.696Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:streamPipe() Flush data to fd:3 successfully. Resolve streamPipe().
2024-03-15T09:22:25.697Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:appendExtent() Write finish, start updating extent metadata. extent:{"id":"342da729-4d82-4c3a-b7bd-0d8a20cfc073","locationId":"test","path":"342da729-4d82-4c3a-b7bd-0d8a20cfc073","size":10,"lastModifiedInMS":1710494545697}
2024-03-15T09:22:25.698Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: FSExtentStore:appendExtent() Update extent metadata done. Resolve()
2024-03-15T09:22:25.698Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue.operate() Job ff19fdb8-252b-4eee-aa64-edcb42ec4421 completes callback, resolve.
2024-03-15T09:22:25.699Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue.operate() Schedule incoming job ee382f60-f4b9-44a3-a5aa-976da35db131
2024-03-15T09:22:25.699Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:1
2024-03-15T09:22:25.699Z 0aeecaef-052f-4754-a894-752c6a986a05 verbose: FSExtentStore:readExtent() Creating read stream. LocationId:test extentId:342da729-4d82-4c3a-b7bd-0d8a20cfc073 path:__test_blob_extent__\342da729-4d82-4c3a-b7bd-0d8a20cfc073 offset:0 count:10 end:9
2024-03-15T09:22:25.700Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue.operate() Job ee382f60-f4b9-44a3-a5aa-976da35db131 completes callback, resolve.
2024-03-15T09:22:25.700Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:10 operations.length:0
2024-03-15T09:22:25.700Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue:execute() return. Operation.length === 0
2024-03-15T09:22:25.701Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:0
2024-03-15T09:22:25.701Z 0aeecaef-052f-4754-a894-752c6a986a05 debug: OperationQueue:execute() return. Operation.length === 0
2024-03-15T09:22:25.705Z 0aeecaef-052f-4754-a894-752c6a986a05 verbose: SerializerMiddleware: Start serializing...
2024-03-15T09:22:25.707Z 0aeecaef-052f-4754-a894-752c6a986a05 info: EndMiddleware: End response. TotalTimeInMS=51 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.29.0","x-ms-client-request-id":"e4147f27-41db-46ae-9312-a4a5deb3206f","x-ms-request-id":"0aeecaef-052f-4754-a894-752c6a986a05","x-ms-version":"2024-02-04","date":"Fri, 15 Mar 2024 09:22:25 GMT","x-ms-request-server-encrypted":"true"}
2024-03-15T09:22:25.709Z 0aeecaef-052f-4754-a894-752c6a986a05 verbose: FSExtentStore:readExtent() Read stream closed. LocationId:test extentId:342da729-4d82-4c3a-b7bd-0d8a20cfc073 path:__test_blob_extent__\342da729-4d82-4c3a-b7bd-0d8a20cfc073 offset:0 count:10 end:9
Maybe you can share client side code you used, so we can try the same code to try to reproduce the issue?
Thanks Emma
Hi @hashim-hivery ,
We found the root cause now.
The block id in your URL is like: blockid=MDAwMDE=
, in my test case, it's blockid=MDAwMDE%3D
. The SDK I used would encode character =
.
From our document, the reserved characters in query parameter value should be encoded: https://learn.microsoft.com/en-us/graph/query-parameters?tabs=http#encoding-query-parameters. =
is one of the reserved characters.
Azurite implementation meets the documentation's requirement.
Thanks Emma
Please feel free to reopen the issue if any further assistant required.
Which service(blob, file, queue, table) does this issue concern?
Blob
Which version of the Azurite was used?
3.29.0
Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)
DockerHub -- mcr.microsoft.com/azure-storage/azurite@sha256:fc94c34a746aee451b7c3a46dda9d7e8844f4edaab5313862fd7ec04519fbbd0
What's the Node.js version?
v14.21.3
What problem was encountered?
Azurite seems to be omitting
blockid
query parameter from the signature calculationSteps to reproduce the issue?
The first line in the below log content obtained from debug mode is the exact failing request.
The "string to sign" used by the server,
Is different from the "string to sign" used in my code:
Log content:
Have you found a mitigation/solution?
No