search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 325 forks source link

SyncCopyFromUri and SyncCopyFromUriAsync always throws Azure.RequestFailedException (Blob Not Found) #767

Open pseudoramble opened 3 years ago

pseudoramble commented 3 years ago

First off, thanks for all the work on this. Azurite has been a nice addition for our local dev. So our team appreciates it quite a bit.

Which service(blob, file, queue, table) does this issue concern?

Blob.

Which version of the Azurite was used?

3.11.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

Visual Studio Code Extension and npm running on Windows.

What's the Node.js version?

15.12.0

What problem was encountered?

Note: This setup is done through a C# app using the Azure.Storage.Blobs library, version 12.8.1.

When attempting to use the SyncCopyFromUri API to move a file from a publicly accessible URL into a container within an Azurite storage account, we continually get a 404 Not Found exception. When executing this against a storage account in Azure, it completes the transfer as expected without an exception.

What I'm unsure of is if Azure is the correct result or if the emulator is the correct result. I'm hoping it's Azure, because that makes this feature incredibly useful. Also for some reason, SyncCopyFromUri computes the MD5 hash server-side for an integrity check, whereas Upload does not seem to. So this additional feature is pretty vital for us.

The exception message:

Azure.RequestFailedException: The specified blob does not exist.
RequestId:...
Time:...
Status: 404 (The specified blob does not exist.)
ErrorCode: BlobNotFound

Headers:
Server: Azurite-Blob/3.11.0
x-ms-error-code: BlobNotFound
x-ms-request-id: ...
Date: ...
Connection: keep-alive
Transfer-Encoding: chunked
Content-Type: application/xml

   at Azure.Storage.Blobs.BlobRestClient.Blob.CopyFromUriAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response)
   at Azure.Storage.Blobs.BlobRestClient.Blob.CopyFromUriAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, Uri copySource, String version, Nullable`1 timeout, IDictionary`2 metadata, Nullable`1 tier, Nullable`1 sourceIfModifiedSince, Nullable`1 sourceIfUnmodifiedSince, Nullable`1 sourceIfMatch, Nullable`1 sourceIfNoneMatch, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 ifMatch, Nullable`1 ifNoneMatch, String ifTags, String leaseId, String requestId, Byte[] sourceContentHash, String blobTagsString, Boolean async, String operationName, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.SyncCopyFromUriInternal(Uri source, IDictionary`2 metadata, IDictionary`2 tags, Nullable`1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Boolean async, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task`1 task)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.SyncCopyFromUri(Uri source, BlobCopyFromUriOptions options, CancellationToken cancellationToken)

Steps to reproduce the issue?

Note: This setup is done through a C# app using the Azure.Storage.Blobs library, version 12.8.1. Again it appears to work fine within Azure directly.

  1. Install Azurite in VSC or in Node (I have not tried it with Docker).
  2. Connect to the storage account. This can be done locally or remotely.
  3. Setup a BlobClient with the name StorageExplorer_signed.zip.
  4. Using the BlobClient instance, execute SyncCopyFromUri or SyncCopyFromUriAsync with a URL that is not in an Azure storage account specifically but is publicly accessible. Example: the Storage Explorer download for OSX will reproduce the issue.

Have you found a mitigation/solution?

The only workaround I can see is to manually do what SyncCopyFromUri seems to do (download locally, upload to the Storage Account, download from the storage account, hash the contents and compare the hashes).

pseudoramble commented 3 years ago

Here are the debug logs from Azurite:

2021-04-28T16:33:33.263Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=https://127.0.0.1/devstoreaccount1/sample/StorageExplorer_signed.zip RequestHeaders:{"host":"127.0.0.1:10000","x-ms-requires-sync":"true","x-ms-copy-source":"https://download.microsoft.com/download/A/E/3/AE32C485-B62B-4437-92F7-8B6B2C48CB40/StorageExplorer_signed.zip","x-ms-version":"2020-04-08","x-ms-client-request-id":"465890de-3257-4eb5-a1bc-a1d4e73c9021","x-ms-return-client-request-id":"true","user-agent":"azsdk-net-Storage.Blobs/12.8.1 (.NET Core 3.1.13; Darwin 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64)","x-ms-date":"Wed, 28 Apr 2021 16:33:33 GMT","authorization":"SharedKey devstoreaccount1:<SHARED KEY HERE>","traceparent":"00-3da4299e475ca741882e8613fd8d0cac-92b6f4167a9aeb43-00","content-length":"0"} ClientIP=127.0.0.1 Protocol=https HTTPVersion=1.1
2021-04-28T16:33:33.263Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info BlobStorageContextMiddleware: Account=devstoreaccount1 Container=sample Blob=StorageExplorer_signed.zip
2021-04-28T16:33:33.263Z b8f462cf-4091-4af2-92a2-5734bcf84b65 verbose DispatchMiddleware: Dispatching request...
2021-04-28T16:33:33.264Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info DispatchMiddleware: Operation=Blob_CopyFromURL
2021-04-28T16:33:33.264Z b8f462cf-4091-4af2-92a2-5734bcf84b65 verbose AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2021-04-28T16:33:33.264Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info PublicAccessAuthenticator:validate() Start validation against public access.
2021-04-28T16:33:33.264Z b8f462cf-4091-4af2-92a2-5734bcf84b65 debug PublicAccessAuthenticator:validate() Getting account properties...
2021-04-28T16:33:33.264Z b8f462cf-4091-4af2-92a2-5734bcf84b65 debug PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: sample, blob: StorageExplorer_signed.zip
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 debug PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container sample
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:465890de-3257-4eb5-a1bc-a1d4e73c9021\nx-ms-copy-source:https://download.microsoft.com/download/A/E/3/AE32C485-B62B-4437-92F7-8B6B2C48CB40/StorageExplorer_signed.zip\nx-ms-date:Wed, 28 Apr 2021 16:33:33 GMT\nx-ms-requires-sync:true\nx-ms-return-client-request-id:true\nx-ms-version:2020-04-08\n/devstoreaccount1/devstoreaccount1/sample/StorageExplorer_signed.zip"
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:<SHARED KEY HERE>
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 verbose DeserializerMiddleware: Start deserializing...
2021-04-28T16:33:33.265Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"requestId":"465890de-3257-4eb5-a1bc-a1d4e73c9021","sourceModifiedAccessConditions":{},"modifiedAccessConditions":{},"leaseAccessConditions":{}},"copySource":"https://download.microsoft.com/download/A/E/3/AE32C485-B62B-4437-92F7-8B6B2C48CB40/StorageExplorer_signed.zip","version":"2020-04-08","xMsRequiresSync":"true"}
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: ErrorName=StorageError ErrorMessage=The specified blob does not exist.  ErrorHTTPStatusCode=404 ErrorHTTPStatusMessage=The specified blob does not exist. ErrorHTTPHeaders={"x-ms-error-code":"BlobNotFound","x-ms-request-id":"b8f462cf-4091-4af2-92a2-5734bcf84b65"} ErrorHTTPBody="<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>BlobNotFound</Code>\n  <Message>The specified blob does not exist.\nRequestId:b8f462cf-4091-4af2-92a2-5734bcf84b65\nTime:2021-04-28T16:33:33.265Z</Message>\n</Error>" ErrorStack="StorageError: The specified blob does not exist.\n\tat Function.getBlobNotFound (c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\dist\\src\\blob\\errors\\StorageErrorFactory.js:29:16)\n\tat BlobHandler.copyFromURL (c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\dist\\src\\blob\\handlers\\BlobHandler.js:504:49)\n\tat c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\dist\\src\\blob\\generated\\middleware\\HandlerMiddlewareFactory.js:58:18\n\tat c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\dist\\src\\blob\\generated\\ExpressMiddlewareFactory.js:77:63\n\tat Layer.handle [as handle_request] (c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\node_modules\\express\\lib\\router\\layer.js:95:5)\n\tat trim_prefix (c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\node_modules\\express\\lib\\router\\index.js:317:13)\n\tat c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\node_modules\\express\\lib\\router\\index.js:284:7\n\tat Function.process_params (c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\node_modules\\express\\lib\\router\\index.js:335:12)\n\tat next (c:\\Users\\User1\\.vscode\\extensions\\azurite.azurite-3.11.0\\node_modules\\express\\lib\\router\\index.js:275:10)\n\tat runMicrotasks (<anonymous>)\n\tat processTicksAndRejections (internal/process/task_queues.js:97:5)"
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Set HTTP code: 404
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Set HTTP status message: The specified blob does not exist.
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Set HTTP Header: x-ms-error-code=BlobNotFound
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Set HTTP Header: x-ms-request-id=b8f462cf-4091-4af2-92a2-5734bcf84b65
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Set content type: application/xml
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 error ErrorMiddleware: Set HTTP body: "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>BlobNotFound</Code>\n  <Message>The specified blob does not exist.\nRequestId:b8f462cf-4091-4af2-92a2-5734bcf84b65\nTime:2021-04-28T16:33:33.265Z</Message>\n</Error>"
2021-04-28T16:33:33.266Z b8f462cf-4091-4af2-92a2-5734bcf84b65 info EndMiddleware: End response. TotalTimeInMS=3 StatusCode=404 StatusMessage=The specified blob does not exist. Headers={"server":"Azurite-Blob/3.11.0","x-ms-error-code":"BlobNotFound","x-ms-request-id":"b8f462cf-4091-4af2-92a2-5734bcf84b65","content-type":"application/xml"}
blueww commented 3 years ago

@pseudoramble Currently Azurite only support blob copy inside same Azurtie instance. You can see it in https://github.com/Azure/Azurite#support-matrix

Copy Blob (Only supports copy within same Azurite instance) Abort Copy Blob (Only supports copy within same Azurite instance) Copy Blob From URL (Only supports copy within same Azurite instance, only on Loki)

Azurite welcome contribution. It would be great if you can raise PR to implement it.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.