search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.77k stars 315 forks source link

Upload from URL not working with a SAS URL #2427

Open pierreinglebert opened 3 weeks ago

pierreinglebert commented 3 weeks ago

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.31

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub

What's the Node.js version?

22

What problem was encountered?

I cannot copy a file from a container to another using uploadFromUrl

Steps to reproduce the issue?

import { BlobSASPermissions, BlobServiceClient, ContainerClient } from "@azure/storage-blob";

const blobServiceClient = BlobServiceClient.fromConnectionString('DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;');
const containerClient_1 = blobServiceClient.getContainerClient('interop-client-1-container');
if (!await containerClient_1.exists()) {
    await containerClient_1.create();
}

const data = 'Hello, World!';
await containerClient_1.getBlockBlobClient('test.txt').upload(data, data.length);
const url = await containerClient_1.getBlockBlobClient('test.txt').generateSasUrl({
    startsOn: new Date(),
    permissions: BlobSASPermissions.parse("racwd"), 
    expiresOn: new Date(new Date().valueOf() + 86400),
});

const containerClient_2 = await blobServiceClient.getContainerClient('interop-container');
await containerClient_2.getBlockBlobClient('copy_test.txt').syncUploadFromURL(url);

If possible, please provide the debug log using the -d parameter, replacing \<pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:

2024-07-11T07:35:39.203Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/interop/019097e9-d379-7e40-b559-e61805d19223 RequestHeaders:{"x-ms-version":"2024-05-04","accept":"application/xml","content-length":"0","x-ms-copy-source":"http://127.0.0.1:10000/devstoreaccount1/interop-client-1-container/test.txt?sv=2024-05-04&st=2024-07-11T07%3A35%3A39Z&se=2024-07-11T07%3A37%3A06Z&sr=b&sp=racwd&sig=YE2q6RRErdlmBLNsqELxXO5bdoYXIgQi0lXo%2FqslIWI%3D","x-ms-blob-type":"BlockBlob","user-agent":"azsdk-js-azure-storage-blob/12.23.0 core-rest-pipeline/1.16.1 Node/20.8.0 OS/(arm64-Darwin-23.5.0)","x-ms-client-request-id":"f4fb964a-eba1-4463-ad37-05fbcfc13bd2","x-ms-date":"Thu, 11 Jul 2024 07:35:39 GMT","authorization":"SharedKey devstoreaccount1:7nZi4knBfVjRZUr9xFdYQhFWCrQvGppY1pDVMuhmfs8=","host":"127.0.0.1:10000","connection":"keep-alive"} ClientIP=172.22.0.1 Protocol=http HTTPVersion=1.1
2024-07-11T07:35:39.203Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=interop Blob=019097e9-d379-7e40-b559-e61805d19223
2024-07-11T07:35:39.203Z e0b2b153-f735-4bf8-97c0-6b72c75399cf verbose: DispatchMiddleware: Dispatching request...
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: DispatchMiddleware: Operation=BlockBlob_PutBlobFromUrl
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: PublicAccessAuthenticator:validate() Start validation against public access.
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf debug: PublicAccessAuthenticator:validate() Getting account properties...
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: interop, blob: 019097e9-d379-7e40-b559-e61805d19223
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container interop
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-client-request-id:f4fb964a-eba1-4463-ad37-05fbcfc13bd2\nx-ms-copy-source:http://127.0.0.1:10000/devstoreaccount1/interop-client-1-container/test.txt?sv=2024-05-04&st=2024-07-11T07%3A35%3A39Z&se=2024-07-11T07%3A37%3A06Z&sr=b&sp=racwd&sig=YE2q6RRErdlmBLNsqELxXO5bdoYXIgQi0lXo%2FqslIWI%3D\nx-ms-date:Thu, 11 Jul 2024 07:35:39 GMT\nx-ms-version:2024-05-04\n/devstoreaccount1/devstoreaccount1/interop/019097e9-d379-7e40-b559-e61805d19223"
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:7nZi4knBfVjRZUr9xFdYQhFWCrQvGppY1pDVMuhmfs8=
2024-07-11T07:35:39.204Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2024-07-11T07:35:39.205Z e0b2b153-f735-4bf8-97c0-6b72c75399cf verbose: DeserializerMiddleware: Start deserializing...
2024-07-11T07:35:39.205Z e0b2b153-f735-4bf8-97c0-6b72c75399cf info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"requestId":"f4fb964a-eba1-4463-ad37-05fbcfc13bd2","blobHTTPHeaders":{},"leaseAccessConditions":{},"cpkInfo":{},"cpkScopeInfo":{},"modifiedAccessConditions":{},"sourceModifiedAccessConditions":{}},"contentLength":0,"version":"2024-05-04","copySource":"http://127.0.0.1:10000/devstoreaccount1/interop-client-1-container/test.txt?sv=2024-05-04&st=2024-07-11T07%3A35%3A39Z&se=2024-07-11T07%3A37%3A06Z&sr=b&sp=racwd&sig=YE2q6RRErdlmBLNsqELxXO5bdoYXIgQi0lXo%2FqslIWI%3D","blobType":"BlockBlob"}
2024-07-11T07:35:39.205Z e0b2b153-f735-4bf8-97c0-6b72c75399cf error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2024-07-11T07:35:39.205Z e0b2b153-f735-4bf8-97c0-6b72c75399cf error: ErrorMiddleware: ErrorName=StorageError ErrorMessage=Current API is not implemented yet. Please vote your wanted features to https://github.com/azure/azurite/issues  ErrorHTTPStatusCode=500 ErrorHTTPStatusMessage=Current API is not implemented yet. Please vote your wanted features to https://github.com/azure/azurite/issues ErrorHTTPHeaders={"x-ms-error-code":"APINotImplemented","x-ms-request-id":"e0b2b153-f735-4bf8-97c0-6b72c75399cf"} ErrorHTTPBody="<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>APINotImplemented</Code>\n  <Message>Current API is not implemented yet. Please vote your wanted features to https://github.com/azure/azurite/issues\nRequestId:e0b2b153-f735-4bf8-97c0-6b72c75399cf\nTime:2024-07-11T07:35:39.205Z</Message>\n</Error>" ErrorStack="StorageError: Current API is not implemented yet. Please vote your wanted features to https://github.com/azure/azurite/issues\n    at BlockBlobHandler.putBlobFromUrl (/opt/azurite/dist/src/blob/handlers/BlockBlobHandler.js:118:15)\n    at /opt/azurite/dist/src/blob/generated/middleware/HandlerMiddlewareFactory.js:58:18\n    at /opt/azurite/dist/src/blob/generated/ExpressMiddlewareFactory.js:78:63\n    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:328:13)\n    at /opt/azurite/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/opt/azurite/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/azurite/node_modules/express/lib/router/index.js:280:10)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
2024-07-11T07:35:39.205Z e0b2b153-f735-4bf8-97c0-6b72c75399cf error: ErrorMiddleware: Set HTTP code: 500
2024-07-11T07:35:39.205Z e0b2b153-f735-4bf8-97c0-6b72c75399cf error: ErrorMiddleware: Set HTTP status message: Current API is not implemented yet. Please vote your wanted features to https://github.com/azure/azurite/issues

Have you found a mitigation/solution?

Not yet

blueww commented 3 weeks ago

@EmmaZhu Would you please help to look?

@pierreinglebert From the error message, the API still not implemented in Azurite.

From the request, we can see there's a header "x-ms-blob-type":"BlockBlob", so this is a "BlockBlob_PutBlobFromUrl" request, which is not implemented in Azurite.

If you just want to copy blob and source/dest blob type are same, you might can try to use Copy blob API instead of PutBlobFromUrl API.

If you really need to use PutBlobFromUrl API, Azurite welcome contribution! It would be great if you could help to raise a PR to implement it!