search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 325 forks source link

PageBlob_UploadPages： Write out-of-range pages should fail #333

Open blueww opened 4 years ago

blueww commented 4 years ago

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.3.0-preview

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

VS code

What's the Node.js version?

v10.15.3

What problem was encountered?

PageBlob_UploadPages： Write out-of-range pages should fail

Steps to reproduce the issue?

Create a page blob with length 4194304 Write a page to 512-4194815 Azurite not fail. Expecte HttpStatusCode: RequestedRangeNotSatisfiable = 416 Expected Error code: "InvalidPageRange"

Have you found a mitigation/solution?

No

Debug Log:

2019-12-18T03:36:34.804Z c39f242d-3972-4fbb-8897-01138ef1de37 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/testcc06e1621d29a4c9eb50b60ddf604c29f/testb11fe8893bb1344c8b80722d9fc679cda RequestHeaders:{"x-ms-blob-type":"PageBlob","x-ms-blob-content-length":"4194304","x-ms-client-request-id":"70fd0739-3169-42a8-bbeb-4fa4607be911","user-agent":"Azure-Storage/11.1.1 (.NET Core; Win32NT 6.2.9200.0)","x-ms-version":"2019-02-02","x-ms-date":"Wed, 18 Dec 2019 03:36:34 GMT","authorization":"SharedKey devstoreaccount1:CNJ5DipGxeZ38/pdQApXz0ZoANlw3lXhl2armyfZ+Pc=","content-length":"0","host":"127.0.0.1:10000"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2019-12-18T03:36:34.805Z c39f242d-3972-4fbb-8897-01138ef1de37 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=testcc06e1621d29a4c9eb50b60ddf604c29f Blob=testb11fe8893bb1344c8b80722d9fc679cda
2019-12-18T03:36:34.806Z c39f242d-3972-4fbb-8897-01138ef1de37 verbose: DispatchMiddleware: Dispatching request...
2019-12-18T03:36:34.808Z c39f242d-3972-4fbb-8897-01138ef1de37 info: DispatchMiddleware: Operation=PageBlob_Create
2019-12-18T03:36:34.808Z c39f242d-3972-4fbb-8897-01138ef1de37 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2019-12-18T03:36:34.809Z c39f242d-3972-4fbb-8897-01138ef1de37 info: PublicAccessAuthenticator:validate() Start validation against public access.
2019-12-18T03:36:34.809Z c39f242d-3972-4fbb-8897-01138ef1de37 debug: PublicAccessAuthenticator:validate() Getting account properties...
2019-12-18T03:36:34.810Z c39f242d-3972-4fbb-8897-01138ef1de37 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: testcc06e1621d29a4c9eb50b60ddf604c29f, blob: testb11fe8893bb1344c8b80722d9fc679cda
2019-12-18T03:36:34.814Z c39f242d-3972-4fbb-8897-01138ef1de37 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container testcc06e1621d29a4c9eb50b60ddf604c29f
2019-12-18T03:36:34.815Z c39f242d-3972-4fbb-8897-01138ef1de37 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2019-12-18T03:36:34.815Z c39f242d-3972-4fbb-8897-01138ef1de37 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-blob-content-length:4194304\nx-ms-blob-type:PageBlob\nx-ms-client-request-id:70fd0739-3169-42a8-bbeb-4fa4607be911\nx-ms-date:Wed, 18 Dec 2019 03:36:34 GMT\nx-ms-version:2019-02-02\n/devstoreaccount1/devstoreaccount1/testcc06e1621d29a4c9eb50b60ddf604c29f/testb11fe8893bb1344c8b80722d9fc679cda"
2019-12-18T03:36:34.815Z c39f242d-3972-4fbb-8897-01138ef1de37 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:CNJ5DipGxeZ38/pdQApXz0ZoANlw3lXhl2armyfZ+Pc=
2019-12-18T03:36:34.816Z c39f242d-3972-4fbb-8897-01138ef1de37 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2019-12-18T03:36:34.816Z c39f242d-3972-4fbb-8897-01138ef1de37 verbose: DeserializerMiddleware: Start deserializing...
2019-12-18T03:36:34.816Z c39f242d-3972-4fbb-8897-01138ef1de37 info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"requestId":"70fd0739-3169-42a8-bbeb-4fa4607be911","blobHTTPHeaders":{},"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"contentLength":0,"blobContentLength":4194304,"version":"2019-02-02","blobType":"PageBlob"}
2019-12-18T03:36:34.818Z c39f242d-3972-4fbb-8897-01138ef1de37 verbose: SerializerMiddleware: Start serializing...
2019-12-18T03:36:34.819Z c39f242d-3972-4fbb-8897-01138ef1de37 info: EndMiddleware: End response. TotalTimeInMS=15 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.3.0-preview","etag":"\"d-gstAftTQg/5N9TV0vkJbTb5weFU\"","last-modified":"Wed, 18 Dec 2019 03:36:34 GMT","x-ms-client-request-id":"70fd0739-3169-42a8-bbeb-4fa4607be911","x-ms-request-id":"c39f242d-3972-4fbb-8897-01138ef1de37","x-ms-version":"2019-02-02","date":"Wed, 18 Dec 2019 03:36:34 GMT","x-ms-request-server-encrypted":"true"}

2019-12-18T03:37:50.662Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/testcc06e1621d29a4c9eb50b60ddf604c29f/testb11fe8893bb1344c8b80722d9fc679cda?comp=page RequestHeaders:{"x-ms-range":"bytes=512-4194815","x-ms-page-write":"Update","x-ms-client-request-id":"6168bc22-aaf8-40e5-b865-e536be30145b","user-agent":"Azure-Storage/11.1.1 (.NET Core; Win32NT 6.2.9200.0)","x-ms-version":"2019-02-02","x-ms-date":"Wed, 18 Dec 2019 03:37:50 GMT","authorization":"SharedKey devstoreaccount1:JR9OPceOu2dQc2LHk658cC0rcVb0M2JlSskvcQm+89g=","content-length":"4194304","host":"127.0.0.1:10000"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2019-12-18T03:37:50.663Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=testcc06e1621d29a4c9eb50b60ddf604c29f Blob=testb11fe8893bb1344c8b80722d9fc679cda
2019-12-18T03:37:50.663Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 verbose: DispatchMiddleware: Dispatching request...
2019-12-18T03:37:50.666Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: DispatchMiddleware: Operation=PageBlob_UploadPages
2019-12-18T03:37:50.666Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2019-12-18T03:37:50.667Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: PublicAccessAuthenticator:validate() Start validation against public access.
2019-12-18T03:37:50.667Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: PublicAccessAuthenticator:validate() Getting account properties...
2019-12-18T03:37:50.667Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: testcc06e1621d29a4c9eb50b60ddf604c29f, blob: testb11fe8893bb1344c8b80722d9fc679cda
2019-12-18T03:37:50.671Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container testcc06e1621d29a4c9eb50b60ddf604c29f
2019-12-18T03:37:50.672Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2019-12-18T03:37:50.672Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n4194304\n\n\n\n\n\n\n\n\nx-ms-client-request-id:6168bc22-aaf8-40e5-b865-e536be30145b\nx-ms-date:Wed, 18 Dec 2019 03:37:50 GMT\nx-ms-page-write:Update\nx-ms-range:bytes=512-4194815\nx-ms-version:2019-02-02\n/devstoreaccount1/devstoreaccount1/testcc06e1621d29a4c9eb50b60ddf604c29f/testb11fe8893bb1344c8b80722d9fc679cda\ncomp:page"
2019-12-18T03:37:50.672Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:JR9OPceOu2dQc2LHk658cC0rcVb0M2JlSskvcQm+89g=
2019-12-18T03:37:50.672Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2019-12-18T03:37:50.674Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 verbose: DeserializerMiddleware: Start deserializing...
2019-12-18T03:37:50.675Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: HandlerMiddleware: DeserializedParameters={"options":{"range":"bytes=512-4194815","requestId":"6168bc22-aaf8-40e5-b865-e536be30145b","leaseAccessConditions":{},"cpkInfo":{},"sequenceNumberAccessConditions":{},"modifiedAccessConditions":{}},"comp":"page","contentLength":4194304,"version":"2019-02-02","pageWrite":"update","body":"ReadableStream"}
2019-12-18T03:37:50.679Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: OperationQueue.operate() Schedule incoming job 77e5ab97-0f60-495d-8058-38aec15f1383
2019-12-18T03:37:50.680Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:10 operations.length:1
2019-12-18T03:37:50.680Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: FSExtentStore:appendExtent() Select extent from idle location for extent append operation. LocationId:1 extentId:bbca821a-16da-4780-849d-bca386400521 offset:4196352 MAX_EXTENT_SIZE:4194304 
2019-12-18T03:37:50.680Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: FSExtentStore:appendExtent() Size of selected extent offset is larger than maximum extent size 4194304 bytes, try appending to new extent. New extent LocationID:1 extentId:a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05 offset:0 MAX_EXTENT_SIZE:4194304 
2019-12-18T03:37:50.681Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: FSExtentStore:appendExtent() Get fd:undefined for extent:a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05 from cache.
2019-12-18T03:37:50.682Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: FSExtentStore:appendExtent() Open file:__blobstorage__\a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05 for extent:a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05, get new fd:53
2019-12-18T03:37:50.683Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: FSExtentStore:appendExtent() Start writing to extent a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05
2019-12-18T03:37:50.705Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: FSExtentStore:appendExtent() Write finish, start updating extent metadata. extent:{"id":"a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05","persistencyId":"Default","path":"a3d48ca1-ee0c-4f94-bf1e-2e8e36170e05","size":4194304,"lastModifiedInMS":1576640270705}
2019-12-18T03:37:50.706Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: FSExtentStore:appendExtent() Update extent metadata done. Resolve()
2019-12-18T03:37:50.706Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: OperationQueue.operate() Job 77e5ab97-0f60-495d-8058-38aec15f1383 completes callback, resolve.
2019-12-18T03:37:50.709Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 verbose: SerializerMiddleware: Start serializing...
2019-12-18T03:37:50.709Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 info: EndMiddleware: End response. TotalTimeInMS=47 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.3.0-preview","etag":"\"d-5nJKsEMuVKolHu9Z9kBQXGr7ZKk\"","last-modified":"Wed, 18 Dec 2019 03:37:50 GMT","x-ms-client-request-id":"6168bc22-aaf8-40e5-b865-e536be30145b","x-ms-request-id":"33bf448d-755b-4560-9d0a-ccb8c46325f1","x-ms-version":"2019-02-02","date":"Wed, 18 Dec 2019 03:37:50 GMT","x-ms-request-server-encrypted":"true"}
2019-12-18T03:37:50.710Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: OperationQueue:execute() Current runningConcurrency:0 maxConcurrency:10 operations.length:0
2019-12-18T03:37:50.710Z 33bf448d-755b-4560-9d0a-ccb8c46325f1 debug: OperationQueue:execute() return. Operation.length === 0

Test Case:

Test Name:  CloudPageBlobWritePages
Test FullName:  Microsoft.Azure.Storage.Blob.CloudPageBlobTest.CloudPageBlobWritePages
Test Source:    C:\code\azure-storage-net\Test\ClassLibraryCommon\Blob\CloudPageBlobTest.cs : line 2063
Test Outcome:   Failed
Test Duration:  0:02:42.94238

Result StackTrace:  
at Microsoft.Azure.Storage.TestHelper.ExpectedException(Action operation, String operationDescription, HttpStatusCode expectedStatusCode, String requestErrorCode) in C:\code\azure-storage-net\Test\ClassLibraryCommon\TestHelper.cs:line 55
   at Microsoft.Azure.Storage.Blob.CloudPageBlobTest.CloudPageBlobWritePages() in C:\code\azure-storage-net\Test\ClassLibraryCommon\Blob\CloudPageBlobTest.cs:line 2094
Result Message: Assert.Fail failed. No Storage exception received while expecting RequestedRangeNotSatisfiable: Writing out-of-range pages should fail
XiaoningLiu commented 4 years ago

Negative check, will update after ga release.

fao89 commented 3 years ago

It is not upload, but it might be related to this issue, when we set a range with starting value greater than the content size we get 206 instead of 416

In [4]: response = requests.get(url, headers={"x-ms-range": "bytes=10985860-10985870"})

In [5]: response.status_code
Out[5]: 206

In [6]: response.headers
Out[6]: {
    'Server': 'Azurite-Blob/3.14.2', 
    'last-modified': 'Fri, 01 Oct 2021 19:10:01 GMT', 
    'x-ms-creation-time': 'Fri, 01 Oct 2021 19:02:06 GMT', 
    'content-length': '-500100', 
    'content-type': 'application/octet-stream', 
    'content-range': 'bytes 10985860-10485759/10485760', 
    'etag': '"0x1CFD59B66D49660"', 
    'content-md5': '1B2M2Y8AsgTpgAmY7PhCfg==', 
    'x-ms-blob-type': 'BlockBlob', 
    'x-ms-lease-state': 'available', 
    'x-ms-lease-status': 'unlocked', 
    'x-ms-request-id': '96f22075-4264-4613-80ee-85d63f4dcbe0', 
    'x-ms-version': '2020-10-02', 
    'accept-ranges': 'bytes', 
    'date': 'Fri, 01 Oct 2021 19:10:44 GMT', 
    'x-ms-server-encrypted': 'true', 
    'Connection': 'keep-alive', 
    'Keep-Alive': 'timeout=5'
}
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

andrewhickman-aveva commented 4 months ago

The error code defined here also doesn't match the service:

https://github.com/Azure/Azurite/blob/9fa8f4a1a5fad9745e92e521f9b58ed561aa100b/src/blob/errors/StorageErrorFactory.ts#L193-L200

I think "Requested Range Not Satisfiable" should be "InvalidPageRange"