search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 325 forks source link

[blob] BlobDownloadOptions: rangeGetContentCrc64 and rangeGetContentMD5 cannot be set at same time. #290

Open ljian3377 opened 4 years ago

ljian3377 commented 4 years ago

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.3.0-preview

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

v12.13.0

What problem was encountered?

Unit test failure: BlobClient.download all parameters set Failed for 

    let exceptionCaught = false;
    try {
      await blobClient.download(2, 1, {
        rangeGetContentMD5: true,
        rangeGetContentCrc64: true
      });
    } catch (err) {
      exceptionCaught = true;
    }
    assert.ok(exceptionCaught);

Here both rangeGetContentMD5 and rangeGetContentCrc64 is specified for the BlobDownloadOptions so we are expecting a failure. Though azurite don't support Crc64 currently, it's up to the design whether to keep this behavior aligned.

debug.log

2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/container157614921951201929/blob157614921955700155 RequestHeaders:{"x-ms-range":"bytes=2-2","x-ms-range-get-content-md5":"true","x-ms-range-get-content-crc64":"true","x-ms-version":"2019-02-02","user-agent":"azsdk-js-storageblob/12.0.1 (NODE-VERSION v12.13.0; Windows_NT 10.0.18362)","x-ms-client-request-id":"f5a8e0f7-6ead-4b18-bab6-274e35b81a9c","x-ms-date":"Thu, 12 Dec 2019 11:13:39 GMT","authorization":"SharedKey devstoreaccount1:y4n7HP1TpAKdwPLxbUyjTNnBdtiZ4OSjmNIjffTYeHk=","cookie":"","accept":"*/*","accept-encoding":"gzip,deflate","host":"127.0.0.1:10000","connection":"keep-alive"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=container157614921951201929 Blob=blob157614921955700155
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: DispatchMiddleware: Dispatching request...
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: DispatchMiddleware: Operation=Blob_Download
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: PublicAccessAuthenticator:validate() Start validation against public access.
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: PublicAccessAuthenticator:validate() Getting account properties...
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: container157614921951201929, blob: blob157614921955700155
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container container157614921951201929
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:f5a8e0f7-6ead-4b18-bab6-274e35b81a9c\nx-ms-date:Thu, 12 Dec 2019 11:13:39 GMT\nx-ms-range:bytes=2-2\nx-ms-range-get-content-crc64:true\nx-ms-range-get-content-md5:true\nx-ms-version:2019-02-02\n/devstoreaccount1/devstoreaccount1/container157614921951201929/blob157614921955700155"
2019-12-12T11:13:39.613Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:y4n7HP1TpAKdwPLxbUyjTNnBdtiZ4OSjmNIjffTYeHk=
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: DeserializerMiddleware: Start deserializing...
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: HandlerMiddleware: DeserializedParameters={"options":{"range":"bytes=2-2","rangeGetContentMD5":true,"rangeGetContentCRC64":true,"requestId":"f5a8e0f7-6ead-4b18-bab6-274e35b81a9c","leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"version":"2019-02-02"}
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: BlobHandler:downloadBlockBlob() NormalizedDownloadRange=bytes=2-2 RequiredContentLength=1
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue.operate() Schedule incoming job fc336ee3-3b70-4901-9b03-0fa41867b497
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:1
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: FSExtentStore:readExtent() Creating read stream. LocationId:Default extentId:b1851f64-d728-4105-94df-3ab4cb27a1fe path:__blobstorage__\b1851f64-d728-4105-94df-3ab4cb27a1fe offset:101 count:1 end:101
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue.operate() Job fc336ee3-3b70-4901-9b03-0fa41867b497 completes callback, resolve.
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:0
2019-12-12T11:13:39.614Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue:execute() return. Operation.length === 0
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue.operate() Schedule incoming job 40ab928c-af31-4ca9-b459-f15361bfbf11
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:1
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: FSExtentStore:readExtent() Creating read stream. LocationId:Default extentId:b1851f64-d728-4105-94df-3ab4cb27a1fe path:__blobstorage__\b1851f64-d728-4105-94df-3ab4cb27a1fe offset:101 count:1 end:101
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue.operate() Job 40ab928c-af31-4ca9-b459-f15361bfbf11 completes callback, resolve.
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: SerializerMiddleware: Start serializing...
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: Serializer: Start returning stream body.
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:100 operations.length:0
2019-12-12T11:13:39.615Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 debug: OperationQueue:execute() return. Operation.length === 0
2019-12-12T11:13:39.616Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: FSExtentStore:readExtent() Read stream closed. LocationId:Default extentId:b1851f64-d728-4105-94df-3ab4cb27a1fe path:__blobstorage__\b1851f64-d728-4105-94df-3ab4cb27a1fe offset:101 count:1 end:101
2019-12-12T11:13:39.617Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 info: EndMiddleware: End response. TotalTimeInMS=4 StatusCode=206 StatusMessage=Partial Content Headers={"server":"Azurite-Blob/3.3.0-preview","last-modified":"Thu, 12 Dec 2019 11:13:39 GMT","content-length":"1","content-type":"application/octet-stream","content-range":"bytes 2-2/11","etag":"\"d-8QGrfKDkXeQWR7XlLW5XgCh3Vk0\"","content-md5":"LblejhqSZ7ehGIVWsgE7Mw==","x-ms-blob-type":"BlockBlob","x-ms-lease-state":"available","x-ms-lease-status":"unlocked","x-ms-client-request-id":"f5a8e0f7-6ead-4b18-bab6-274e35b81a9c","x-ms-request-id":"9e3e42c4-c1dd-487f-b701-5e9c9c73abf1","x-ms-version":"2019-02-02","date":"Thu, 12 Dec 2019 11:13:39 GMT","x-ms-server-encrypted":"true","x-ms-blob-content-md5":"sQqNsWTgdUEFt6mb5y4/5Q=="}
2019-12-12T11:13:39.617Z 9e3e42c4-c1dd-487f-b701-5e9c9c73abf1 verbose: FSExtentStore:readExtent() Read stream closed. LocationId:Default extentId:b1851f64-d728-4105-94df-3ab4cb27a1fe path:__blobstorage__\b1851f64-d728-4105-94df-3ab4cb27a1fe offset:101 count:1 end:101

request_id

f5a8e0f7-6ead-4b18-bab6-274e35b81a9c

ljian3377 commented 4 years ago

@XiaoningLiu

XiaoningLiu commented 4 years ago

Thanks @ljian3377 ! Azurite doesn't support CRC64, so it just ignore the header. @blueww Can we return a not supported error when customer sets CRC64 header?