search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.8k stars 320 forks source link

It doesn't return an error when upload block blob with wrong blobContentMD5 in Azurite #822

Open zzhxiaofeng opened 3 years ago

zzhxiaofeng commented 3 years ago

Error Description: In Azure, it will return an error Md5Mismatch if the uploaded md5 value is wrong when upload block blob. But in Azurite. it will upload block blob successfully. Azure error is shown as following: Screenshot 2021-06-08 185933

To Reproduce: Please run the demo code as following:

import { BlobServiceClient, StorageSharedKeyCredential } from "@azure/storage-blob";
import * as crypto from "crypto";

async function main(){
  const blobServiceClient= new BlobServiceClient("<Azurite-https-endpoint>",new StorageSharedKeyCredential("<account-name>","<account-key>"));
  const containerClient = blobServiceClient.getContainerClient("<container-name>");
  await containerClient.create();
  const content = "correct value";
  const blockBlobClient = containerClient.getBlockBlobClient("<blob-name>");
  const sourceContentMD5 = crypto
  .createHash("md5")
  .update(Buffer.from("wrong value"))
  .digest();
  const uploadBlobResponse = await blockBlobClient.upload(content, Buffer.byteLength(content),{blobHTTPHeaders:{blobContentMD5:sourceContentMD5}});
  console.log(`Upload block blob ${blobName} successfully`, uploadBlobResponse.requestId);
}

main().catch((err)=>{
  console.log(err);
});

Error Track: The process code of md5 is shown as following: https://github.com/Azure/Azurite/blob/master/src/blob/handlers/BlockBlobHandler.ts#L46 image image The request doesn't contain content-md5 when upload a block blob via put blob API. So as shown, it doesn't use uploaded md5 value to be compared with the md5 value calculated by the server and not return any error.

Expected Behavior: It should return an error Md5Mismatch when uploaded md5 value is wrong.

@jongio for notification.

blueww commented 3 years ago

@zzhxiaofeng

Thanks for raising the issue! Could you please add the Azurite debug log to help investigation?

As you said "The request doesn't contain content-md5", so how does user input MD5 with the put blob API?

zzhxiaofeng commented 3 years ago

@blueww Ok, Azurite debug log is shown as following:

2021-06-09T06:05:33.285Z     info: Azurite Blob service is starting on 127.0.0.1:10000
2021-06-09T06:05:33.286Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:05:33.286Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:05:33.418Z     info: BlobGCManager:start() Starting BlobGCManager. Set status to Initializing.
2021-06-09T06:05:33.419Z     info: BlobGCManager:start() Trigger mark and sweep loop. Set status to Running.
2021-06-09T06:05:33.419Z     info: BlobGCManager:markSweepLoop() Start next mark and sweep.
2021-06-09T06:05:33.419Z     info: BlobGCManager:markSweep() Get all extents.
2021-06-09T06:05:33.420Z     info: BlobGCManager:start() BlobGCManager successfully started.
2021-06-09T06:05:33.428Z     info: BlobGCManager:markSweep() Got 1 extents.
2021-06-09T06:05:33.428Z     info: BlobGCManager:markSweep() Get referred extents.
2021-06-09T06:05:33.429Z     info: BlobGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:05:33.429Z     info: BlobGCManager:markSweepLoop() Mark and sweep finished, taken 10ms.
2021-06-09T06:05:33.429Z     info: BlobGCManager:markSweepLoop() Sleep for 600000ms.
2021-06-09T06:05:33.431Z     info: Azurite Blob service successfully listens on https://127.0.0.1:10000
2021-06-09T06:05:33.432Z     info: Azurite Queue service is starting on 127.0.0.1:10001
2021-06-09T06:05:33.432Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:05:33.432Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:05:33.494Z     info: QueueGCManager:start() Starting QueueGCManager, set status to Initializing
2021-06-09T06:05:33.494Z     info: QueueGCManager:start() Trigger mark and sweep loop, set status to Running.
2021-06-09T06:05:33.494Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:05:33.495Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:05:33.495Z     info: QueueGCManager:start() QueueGCManager successfully started.
2021-06-09T06:05:33.495Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:05:33.495Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:05:33.495Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:05:33.495Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms.
2021-06-09T06:05:33.495Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:05:33.496Z     info: Azurite Queue service successfully listens on https://127.0.0.1:10001
2021-06-09T06:05:33.496Z     info: Azurite Table service is starting on 127.0.0.1:10002
2021-06-09T06:05:33.497Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:05:33.497Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:05:33.525Z     info: Azurite Table service successfully listens on https://127.0.0.1:10002
2021-06-09T06:06:33.295Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:06:33.295Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:06:33.437Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:06:33.437Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:06:33.501Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:06:33.501Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:06:33.502Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:06:33.502Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:06:33.502Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:06:33.502Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms.
2021-06-09T06:06:33.502Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:06:33.502Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:06:33.502Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:07:33.301Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:07:33.302Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:07:33.442Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:07:33.443Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:07:33.505Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:07:33.505Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:07:33.505Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:07:33.505Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:07:33.506Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:07:33.506Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms.
2021-06-09T06:07:33.506Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:07:33.506Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:07:33.506Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:08:33.315Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:08:33.315Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:08:33.454Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:08:33.454Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:08:33.517Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:08:33.517Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:08:33.517Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:08:33.517Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:08:33.517Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:08:33.517Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 0ms.
2021-06-09T06:08:33.517Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:08:33.517Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:08:33.517Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:09:33.322Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:09:33.322Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:09:33.462Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:09:33.462Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:09:33.524Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:09:33.524Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:09:33.524Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:09:33.524Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:09:33.525Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:09:33.525Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms.
2021-06-09T06:09:33.525Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:09:33.525Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:09:33.525Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:10:33.327Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:10:33.327Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:10:33.467Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:10:33.468Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:10:33.537Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:10:33.537Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:10:33.537Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:10:33.537Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:10:33.537Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:10:33.537Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 0ms.
2021-06-09T06:10:33.537Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:10:33.537Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:10:33.537Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:11:12.421Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=https://127.0.0.1/devstoreaccount1/newcontainer1623219072343?restype=container RequestHeaders:{"x-ms-version":"2020-06-12","user-agent":"azsdk-js-storageblob/12.5.0 (NODE-VERSION v12.18.0; Windows_NT 10.0.19043)","x-ms-client-request-id":"635bb7ae-d369-476a-bb00-4e915629bc93","x-ms-date":"Wed, 09 Jun 2021 06:11:12 GMT","authorization":"SharedKey devstoreaccount1:W12awBelJ0E1N7qFrq9NYNBl+6yu/P3EXRFHPEBfpdc=","cookie":"","accept":"*/*","content-length":"0","host":"127.0.0.1:10000","connection":"keep-alive"} ClientIP=127.0.0.1 Protocol=https HTTPVersion=1.1
2021-06-09T06:11:12.422Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=newcontainer1623219072343 Blob=
2021-06-09T06:11:12.422Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 verbose: DispatchMiddleware: Dispatching request...
2021-06-09T06:11:12.423Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: DispatchMiddleware: Operation=Container_Create
2021-06-09T06:11:12.424Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2021-06-09T06:11:12.424Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: PublicAccessAuthenticator:validate() Start validation against public access.
2021-06-09T06:11:12.424Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 debug: PublicAccessAuthenticator:validate() Getting account properties...
2021-06-09T06:11:12.424Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: newcontainer1623219072343, blob: 
2021-06-09T06:11:12.499Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container newcontainer1623219072343
2021-06-09T06:11:12.499Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2021-06-09T06:11:12.500Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:635bb7ae-d369-476a-bb00-4e915629bc93\nx-ms-date:Wed, 09 Jun 2021 06:11:12 GMT\nx-ms-version:2020-06-12\n/devstoreaccount1/devstoreaccount1/newcontainer1623219072343\nrestype:container"
2021-06-09T06:11:12.501Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:W12awBelJ0E1N7qFrq9NYNBl+6yu/P3EXRFHPEBfpdc=
2021-06-09T06:11:12.501Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2021-06-09T06:11:12.501Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 verbose: DeserializerMiddleware: Start deserializing...
2021-06-09T06:11:12.502Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"requestId":"635bb7ae-d369-476a-bb00-4e915629bc93"},"restype":"container","version":"2020-06-12"}
2021-06-09T06:11:12.504Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 verbose: SerializerMiddleware: Start serializing...
2021-06-09T06:11:12.505Z 6bf9cf15-cec9-4705-bf61-ee102a0f2b78 info: EndMiddleware: End response. TotalTimeInMS=84 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.12.0","etag":"\"0x1DC732F7366BEB0\"","last-modified":"Wed, 09 Jun 2021 06:11:12 GMT","x-ms-client-request-id":"635bb7ae-d369-476a-bb00-4e915629bc93","x-ms-request-id":"6bf9cf15-cec9-4705-bf61-ee102a0f2b78","x-ms-version":"2020-06-12"}
2021-06-09T06:11:12.515Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=https://127.0.0.1/devstoreaccount1/newcontainer1623219072343/newblob1623219072503 RequestHeaders:{"content-type":"application/octet-stream","content-length":"13","x-ms-version":"2020-06-12","x-ms-blob-type":"BlockBlob","x-ms-blob-content-md5":"VmS4b+CousSJaVl8Way/aA==","user-agent":"azsdk-js-storageblob/12.5.0 (NODE-VERSION v12.18.0; Windows_NT 10.0.19043)","x-ms-client-request-id":"44994b58-2af2-4517-b163-2f41f2410a08","x-ms-date":"Wed, 09 Jun 2021 06:11:12 GMT","authorization":"SharedKey devstoreaccount1:hFyP1PYRtQHbRnbnkFrDIjFtRRbBY7f6CVx+9YXB40I=","cookie":"","accept":"*/*","host":"127.0.0.1:10000","connection":"keep-alive"} ClientIP=127.0.0.1 Protocol=https HTTPVersion=1.1
2021-06-09T06:11:12.515Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=newcontainer1623219072343 Blob=newblob1623219072503
2021-06-09T06:11:12.515Z c5b2a997-063d-4a1d-b6da-1ce161663013 verbose: DispatchMiddleware: Dispatching request...
2021-06-09T06:11:12.517Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: DispatchMiddleware: Operation=BlockBlob_Upload
2021-06-09T06:11:12.517Z c5b2a997-063d-4a1d-b6da-1ce161663013 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2021-06-09T06:11:12.517Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: PublicAccessAuthenticator:validate() Start validation against public access.
2021-06-09T06:11:12.517Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: PublicAccessAuthenticator:validate() Getting account properties...
2021-06-09T06:11:12.517Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: newcontainer1623219072343, blob: newblob1623219072503
2021-06-09T06:11:12.518Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container newcontainer1623219072343
2021-06-09T06:11:12.518Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2021-06-09T06:11:12.518Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n13\n\napplication/octet-stream\n\n\n\n\n\n\nx-ms-blob-content-md5:VmS4b+CousSJaVl8Way/aA==\nx-ms-blob-type:BlockBlob\nx-ms-client-request-id:44994b58-2af2-4517-b163-2f41f2410a08\nx-ms-date:Wed, 09 Jun 2021 06:11:12 GMT\nx-ms-version:2020-06-12\n/devstoreaccount1/devstoreaccount1/newcontainer1623219072343/newblob1623219072503"
2021-06-09T06:11:12.518Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:hFyP1PYRtQHbRnbnkFrDIjFtRRbBY7f6CVx+9YXB40I=
2021-06-09T06:11:12.518Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2021-06-09T06:11:12.518Z c5b2a997-063d-4a1d-b6da-1ce161663013 verbose: DeserializerMiddleware: Start deserializing...
2021-06-09T06:11:12.519Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"requestId":"44994b58-2af2-4517-b163-2f41f2410a08","blobHTTPHeaders":{"blobContentMD5":{"type":"Buffer","data":[86,100,184,111,224,168,186,196,137,105,89,124,89,172,191,104]}},"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"contentLength":13,"version":"2020-06-12","blobType":"BlockBlob","body":"ReadableStream"}
2021-06-09T06:11:12.519Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue.operate() Schedule incoming job c02be3b2-2c6e-4d52-823b-2cf244267242
2021-06-09T06:11:12.519Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:50 operations.length:1
2021-06-09T06:11:12.520Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: FSExtentStore:appendExtent() Select extent from idle location for extent append operation. LocationId:1 extentId:237ec147-43fd-4086-86ea-c33f24101bec offset:0 MAX_EXTENT_SIZE:67108864 
2021-06-09T06:11:12.520Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:appendExtent() Get fd:undefined for extent:237ec147-43fd-4086-86ea-c33f24101bec from cache.
2021-06-09T06:11:12.520Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:appendExtent() Open file:E:\cc\Azurite\__blobstorage__\237ec147-43fd-4086-86ea-c33f24101bec for extent:237ec147-43fd-4086-86ea-c33f24101bec, get new fd:4
2021-06-09T06:11:12.521Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:appendExtent() Created write stream for fd:4
2021-06-09T06:11:12.521Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:appendExtent() Start writing to extent 237ec147-43fd-4086-86ea-c33f24101bec
2021-06-09T06:11:12.521Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:streamPipe() Start piping data to write stream
2021-06-09T06:11:12.521Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:streamPipe() Readable stream triggers close event, 13 bytes piped
2021-06-09T06:11:12.521Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:streamPipe() Invoke write stream end()
2021-06-09T06:11:12.521Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:streamPipe() Writable stream triggers finish event, after 13 bytes piped. Flush data to fd:4.
2021-06-09T06:11:12.565Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:streamPipe() Flush data to fd:4 successfully. Resolve streamPipe().
2021-06-09T06:11:12.566Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:appendExtent() Write finish, start updating extent metadata. extent:{"id":"237ec147-43fd-4086-86ea-c33f24101bec","locationId":"Default","path":"237ec147-43fd-4086-86ea-c33f24101bec","size":13,"lastModifiedInMS":1623219072566}
2021-06-09T06:11:12.566Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:appendExtent() Update extent metadata done. Resolve()
2021-06-09T06:11:12.566Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue.operate() Job c02be3b2-2c6e-4d52-823b-2cf244267242 completes callback, resolve.
2021-06-09T06:11:12.566Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue.operate() Schedule incoming job 072ce337-1204-4aa7-a4f7-edf28e64df4a
2021-06-09T06:11:12.566Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:1
2021-06-09T06:11:12.566Z c5b2a997-063d-4a1d-b6da-1ce161663013 verbose: FSExtentStore:readExtent() Creating read stream. LocationId:Default extentId:237ec147-43fd-4086-86ea-c33f24101bec path:E:\cc\Azurite\__blobstorage__\237ec147-43fd-4086-86ea-c33f24101bec offset:0 count:13 end:12
2021-06-09T06:11:12.567Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue.operate() Job 072ce337-1204-4aa7-a4f7-edf28e64df4a completes callback, resolve.
2021-06-09T06:11:12.567Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:50 operations.length:0
2021-06-09T06:11:12.568Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue:execute() return. Operation.length === 0
2021-06-09T06:11:12.568Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:0
2021-06-09T06:11:12.568Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: OperationQueue:execute() return. Operation.length === 0
2021-06-09T06:11:12.569Z c5b2a997-063d-4a1d-b6da-1ce161663013 verbose: SerializerMiddleware: Start serializing...
2021-06-09T06:11:12.569Z c5b2a997-063d-4a1d-b6da-1ce161663013 info: EndMiddleware: End response. TotalTimeInMS=54 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.12.0","etag":"\"0x20333FB9A0464E0\"","last-modified":"Wed, 09 Jun 2021 06:11:12 GMT","content-md5":"6pB2Y1wsrAyUKHI4lHySpg==","x-ms-client-request-id":"44994b58-2af2-4517-b163-2f41f2410a08","x-ms-request-id":"c5b2a997-063d-4a1d-b6da-1ce161663013","x-ms-version":"2020-06-12","date":"Wed, 09 Jun 2021 06:11:12 GMT","x-ms-request-server-encrypted":"true"}
2021-06-09T06:11:12.570Z c5b2a997-063d-4a1d-b6da-1ce161663013 debug: FSExtentStore:streamPipe() Readable stream triggers close event, 13 bytes piped
2021-06-09T06:11:12.570Z c5b2a997-063d-4a1d-b6da-1ce161663013 verbose: FSExtentStore:readExtent() Read stream closed. LocationId:Default extentId:237ec147-43fd-4086-86ea-c33f24101bec path:E:\cc\Azurite\__blobstorage__\237ec147-43fd-4086-86ea-c33f24101bec offset:0 count:13 end:12
2021-06-09T06:11:33.329Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:11:33.329Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:11:33.468Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:11:33.468Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:11:33.546Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:11:33.547Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:11:33.547Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:11:33.547Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:11:33.547Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:11:33.547Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 0ms.
2021-06-09T06:11:33.547Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:11:33.547Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:11:33.547Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:12:33.336Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:12:33.336Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:12:33.495Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:12:33.495Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-06-09T06:12:33.574Z     info: QueueGCManager:markSweepLoop() Start new mark and sweep.
2021-06-09T06:12:33.574Z     info: QueueGCManger:markSweep() Get all extents.
2021-06-09T06:12:33.574Z     info: QueueGCManager:marksweep() Get 0 extents.
2021-06-09T06:12:33.574Z     info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
2021-06-09T06:12:33.574Z     info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2021-06-09T06:12:33.575Z     info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms.
2021-06-09T06:12:33.575Z     info: QueueGCManager:markSweepLoop() Sleep for 60000
2021-06-09T06:12:33.575Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-06-09T06:12:33.575Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.

There are 2 ways to upload a block blob.

  1. Via single Put Blob API
  2. Multi Stage Block + One Commit Block List API call.

User input MD5 with the put blob API just like I do in my demo code. We can't get content-md5 in any case when use put blob API. So when use the first way, we don't need to check the content-md5 value and it needs to modify the code of Azurite server to make sure we can get uploaded md5 value and md5 value calculated by the server to compare if content-md5 is null or undefined.

blueww commented 3 years ago

@zzhxiaofeng

Thanks for the debug log!

To clarify the requested change from you, in Put Block Blob API:

  1. Azurite should fail the request, when the input content MD5 is not aligned with the content.
  2. Azurite should calculate the content MD5 and add to blob property, when user don't input content MD5.

Is my understanding right?

zzhxiaofeng commented 3 years ago

@blueww The first one maybe is right. but the second is wrong. Let me talk about it in detail.

  1. When in Put Block Blob API, request doesn't contain content-md5 no matter what value you input. User input the md5 value as blob-content-md5 to compare with md5 value calculated from uploaded content by the server.
  2. It checks content-md5 in Azurite as my picture shown. So it is bug of Azurite and it needs to modify.
blueww commented 3 years ago

@zzhxiaofeng

So do you mean following? Azurite should not only check the MD5 input with header content-md5, but also MD5 input with header x-ms-blob-content-md5.

If so, we will evaluate it and fix in a future release.

zzhxiaofeng commented 3 years ago

@blueww I just suggest that content-md5 can’t need to be checked in put block blob API in Azurite. Or set content-md5 to be the same as x-ms-blob-content-md5 before checking if content-md5 is undefined.

blueww commented 3 years ago

@zzhxiaofeng As we discussed offline, this issue is for: Azurite should also check x-ms-blob-content-md5 aligned with blob content in put blob API, to be aligned with Azure server.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.