Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.74k stars 309 forks source link

[BUG] All requests to Azurite in testcontainers container return 400. #2369

Closed avivunitq closed 4 months ago

avivunitq commented 4 months ago

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.29.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub (through TestContainers GenericContainer)

What's the Node.js version?

whatever is included with the dockerfile

What problem was encountered?

I have set up access to Azure Blob Storage in Java 21 using Spring Cloud Azure 5.9.1. See here: https://learn.microsoft.com/en-us/azure/developer/java/spring-framework/resource-handling

Everything is working as expected. I am now trying to integration test my code. Unfortunately, Azure does not have an easy way to set up a container with Testcontainers, so I set one up with

@Container
  static GenericContainer<?> azureBlobStorageContainer = new GenericContainer<>(
      DockerImageName.parse("mcr.microsoft.com/azure-storage/azurite")
  )

However, now when my code accesses the blob storage resource, it encounters an error.

Resource resource = resourceLoader.getResource(String.format("azure-blob://%s/%s", azureBlobStorageContainer, fileName));
2024-02-27 16:17:30 172.17.0.1 - - [27/Feb/2024:21:17:30 +0000] "PUT /test-container?restype=container HTTP/1.1" 400 -

Looking in the debug log, I see this error:

2024-02-27T21:17:30.768Z 705e3745-3759-4ad0-ac05-17408e96aeea info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://localhost/test-container?restype=container RequestHeaders:{"host":"localhost:53880","date":"Tue, 27 Feb 2024 21:17:30 GMT","authorization":"SharedKey devstoreaccount1:u5KEwd3hQqzampyjROwnRBx3uodkdS7DX/nYOQ/ewkQ=","x-ms-version":"2023-11-03","x-ms-client-request-id":"a19071b3-273d-4841-b148-a3483486499d","accept":"application/xml","user-agent":"az-sp-sb/5.9.1 azsdk-java-azure-storage-blob/12.25.1 (21.0.2; Mac OS X; 14.2.1)","content-length":"0"} ClientIP=172.17.0.1 Protocol=http HTTPVersion=1.1
2024-02-27T21:17:30.769Z 705e3745-3759-4ad0-ac05-17408e96aeea info: BlobStorageContextMiddleware: Account=test-container Container=undefined Blob=
2024-02-27T21:17:30.769Z 705e3745-3759-4ad0-ac05-17408e96aeea verbose: DispatchMiddleware: Dispatching request...
2024-02-27T21:17:30.770Z 705e3745-3759-4ad0-ac05-17408e96aeea error: DispatchMiddleware: Incoming URL doesn't match any of swagger defined request patterns.
2024-02-27T21:17:30.770Z 705e3745-3759-4ad0-ac05-17408e96aeea error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2024-02-27T21:17:30.771Z 705e3745-3759-4ad0-ac05-17408e96aeea error: ErrorMiddleware: ErrorName=UnsupportedRequestError ErrorMessage=Incoming URL doesn't match any of swagger defined request patterns.  ErrorHTTPStatusCode=400 ErrorHTTPStatusMessage=undefined ErrorHTTPHeaders=undefined ErrorHTTPBody=undefined ErrorStack="UnsupportedRequestError: Incoming URL doesn't match any of swagger defined request patterns.\n    at dispatchMiddleware (/opt/azurite/dist/src/blob/generated/middleware/dispatch.middleware.js:41:30)\n    at /opt/azurite/dist/src/blob/generated/ExpressMiddlewareFactory.js:50:47\n    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:328:13)\n    at /opt/azurite/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/opt/azurite/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/azurite/node_modules/express/lib/router/index.js:280:10)\n    at blobStorageContextMiddleware (/opt/azurite/dist/src/blob/middlewares/blobStorageContext.middleware.js:137:5)\n    at /opt/azurite/dist/src/blob/middlewares/blobStorageContext.middleware.js:15:16\n    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)"
2024-02-27T21:17:30.771Z 705e3745-3759-4ad0-ac05-17408e96aeea error: ErrorMiddleware: Set HTTP code: 400
2024-02-27T21:17:30.771Z 705e3745-3759-4ad0-ac05-17408e96aeea error: ErrorMiddleware: Set HTTP body: undefined
2024-02-27T21:17:30.771Z 705e3745-3759-4ad0-ac05-17408e96aeea info: EndMiddleware: End response. TotalTimeInMS=3 StatusCode=400 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.29.0"}

when I try to replicate this request using curl, I receive the same error.

I am using the following spring cloud azure blob storage configuration parameters, including the default dev storage account credentials. I followed the instructions here and added a connection string as well: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=visual-studio%2Cblob-storage#connection-strings

    // spring cloud azure config
    registry.add(
        "spring.cloud.azure.storage.blob.endpoint",
        () -> String.format("http://localhost:%s", azureBlobStorageContainer.getFirstMappedPort())
    );

    // azurite key and account name taken from here: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=docker-hub%2Cblob-storage#well-known-storage-account-and-key
    registry.add(
        "spring.cloud.azure.storage.blob.account-name",
        () -> "devstoreaccount1"
    );
    registry.add(
        "spring.cloud.azure.storage.blob.account-key",
        () -> "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
    );

    registry.add(
        "spring.cloud.azure.storage.blob.connection-string",
        () -> "UseDevelopmentStorage=true"
    );

I have not been able to successfully make any calls to the Azurite container--it seems they all return 400.

Steps to reproduce the issue?

Launch Azurite with Testcontainers. Try to connect with Spring Cloud Azure, or directly through a cURL request to the REST API.

If possible, please provide the debug log using the -d parameter, replacing \<pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:

-d "/logs/debug.log"

Please be sure to remove any PII or sensitive information before sharing!
The debug log will log raw request headers and bodies, so that we can replay these against Azurite using REST and create tests to validate resolution.

Have you found a mitigation/solution?

No.

blueww commented 4 months ago

@avivunitq

The incoming request Uri missing the storage account name: RequestURL=http://localhost/test-container?restype=container If you use localhost, the Uri should look like RequestURL=http://localhost/devstoreaccount1/test-container?restype=container You can find more details in https://github.com/Azure/Azurite?tab=readme-ov-file#endpoint--connection-url

avivunitq commented 4 months ago

@avivunitq

The incoming request Uri missing the storage account name: RequestURL=http://localhost/test-container?restype=container If you use localhost, the Uri should look like RequestURL=http://localhost/devstoreaccount1/test-container?restype=container You can find more details in https://github.com/Azure/Azurite?tab=readme-ov-file#endpoint--connection-url

I tried this as well, but then I got an 403 authentication error. But really shouldn't it work as stated? The account key, endpoint, and name should configure all of that behind the scenes, shouldn't it?

avivunitq commented 4 months ago

Ah, it worked! With REST it wasn't working, but changing the config did the trick. I changed the configured endpoint from this:

// spring cloud azure config
registry.add(
    "spring.cloud.azure.storage.blob.endpoint",
    () -> String.format("http://localhost:%s", azureBlobStorageContainer.getFirstMappedPort())
);

to this:

    // spring cloud azure config
    registry.add(
        "spring.cloud.azure.storage.blob.endpoint",
        () -> String.format("http://localhost:%s/devstoreaccount1/", azureBlobStorageContainer.getFirstMappedPort())
    );
blueww commented 4 months ago

@avivunitq

Good to know your scenario works!

BTW, you can find the sample connecting string for Azurite in https://github.com/Azure/Azurite?tab=readme-ov-file#https-connection-strings Please note, the account name should be included in endpoint like : BlobEndpoint=https://127.0.0.1:10000/devstoreaccount1

I will close this issue. Feel free to contact us again if you need any further assistance.