search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 324 forks source link

Azurite SAS query token access to blob file #1263

Open hholst80 opened 2 years ago

hholst80 commented 2 years ago

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

$ docker run --rm mcr.microsoft.com/azure-storage/azurite:latest azurite --version           
3.15.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

mcr.microsoft.com/azure-storage/azurite:latest

What's the Node.js version?

/opt/azurite # node --version
v14.17.0

What problem was encountered?

SAS token access does not work when the connect string references service endpoints not running on localhost, as per default.

Steps to reproduce the issue?

I am using two separate containers, one called "azurite" running azurite, and one called "func" that hosts the local Azure Function App development environment. If I run the function app stack on the host directly and use a connection string that references localhost (and export those ports as per documentation on Azurite) it works.

The following code will give me an URL that I can access in Azure, but Azurite will give a 403.

    import { BlobClient, BlobSASPermissions } from '@azure/storage-blob'

    try {
        const blobName = "input.json"
        const client = new BlobClient(connectionString, "invoices", blobName)
        const options = {
            permissions: BlobSASPermissions.from({ read: true }),
            expiresOn: new Date(Date.now() + 24 * 3600 * 1000),
            // startsOn: new Date(Date.now() - 300 * 1000),
            // protocol: SASProtocol.HttpsAndHttp,
        }
        //const url = (await client.generateSasUrl(options)).replace(/%2F/g,"/")
        const url = await client.generateSasUrl(options)
        context.log(url)
    }
    catch (err) {
        console.log(err)
    }

Have you found a mitigation/solution?

No

blueww commented 2 years ago

@hholst80 Would you please share the Azurite debug log for this issue? And please also share the command you use to start Azurite.

hholst80 commented 2 years ago

I think it is related to the AZURE_STORAGE_CONNECTION_STRING used. I run my function code from inside a container. I tried to repro the issue on my windows pc here (where I actually have all the tooling installed on the host machine) and there it works. I think it should work because the function code itself works, it can use the connection string provided, but where the localhost:10000 is replaced by the container hostname azurite:10000. Let me try it out again back home on my regular workstation later today.

hholst80 commented 2 years ago

Here is the relevant part of the logs from my Linux workstation where I am running the stack in a container environment. That is, both Azurite and Azure Function app is running as two separate containers.

Azurite is started as

  azurite:
    user: "0" # in case someone starts thinking of "best practices" and changes this to a custom user.
    container_name: azurite
    hostname: azurite
    image: mcr.microsoft.com/azure-storage/azurite:latest
    command: azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 -d /tmp/debug.log
    ports:
      - 10000-10002:10000-10002
    working_dir: /azurite
    volumes:
      - azurite:/azurite

The connection string used from the function app is

export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite:10000/devstoreaccount1;QueueEndpoint=http://azurite:10001/devstoreaccount1;TableEndpoint=http://azurite:10002/devstoreaccount1;"

which is also defined accordingly in local.settings.json

{
  "IsEncrypted": false,
  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite:10000/devstoreaccount1;QueueEndpoint=http://azurite:10001/devstoreaccount1;TableEndpoint=http://azurite:10002/devstoreaccount1;"
  }
}

Q: Do I need to generate a new AccountKey if I do not access azurite via "localhost"?

/opt/azurite # grep "validation failed." /tmp/debug.log -C15
2022-01-05T12:21:14.387Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: AccountSASAuthenticator:validate() Getting account properties...
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: AccountSASAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: invoices, blob: input.json
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: AccountSASAuthenticator:validate() Got account properties successfully.
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: AccountSASAuthenticator:validate() Retrieved signature from URL parameter sig: XumrafwEGOSjqUVsmq4OVwZhhtF6nMxxZbDFoFouPgs=
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 info: AccountSASAuthenticator:validate() Failed to get valid account SAS values from request.
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 info: BlobSASAuthenticator:validate() Start validation against blob service Shared Access Signature pattern.
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Getting account properties...
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: invoices, blob: input.json
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Got account properties successfully.
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Retrieved signature from URL parameter sig: XumrafwEGOSjqUVsmq4OVwZhhtF6nMxxZbDFoFouPgs=
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Signed resource type is b.
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Successfully got valid blob service SAS values from request. {"version":"2020-10-02","protocol":"https,http","startTime":"2022-01-05T11:55:05Z","expiryTime":"2022-01-06T12:00:05Z","permissions":"r","containerName":"invoices","blobName":"input.json","signedResource":"b"}
2022-01-05T12:21:14.388Z 6c4b9643-4050-4b51-84de-ba11e1636e36 info: BlobSASAuthenticator:validate() Validate signature based account key1.
2022-01-05T12:21:14.389Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() String to sign is: "r\n2022-01-05T11:55:05Z\n2022-01-06T12:00:05Z\n/blob/devstoreaccount1/invoices/input.json\n\n\nhttps,http\n2020-10-02\nb\n\n\n\n\n\n"
2022-01-05T12:21:14.389Z 6c4b9643-4050-4b51-84de-ba11e1636e36 debug: BlobSASAuthenticator:validate() Calculated signature is: qkUEt42vStwAAIW7pr7Q3/RR13YipDarFTAqbwRDn3E=
2022-01-05T12:21:14.389Z 6c4b9643-4050-4b51-84de-ba11e1636e36 info: BlobSASAuthenticator:validate() Signature based on key1 validation failed.
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: ErrorName=StorageError ErrorMessage=Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.  ErrorHTTPStatusCode=403 ErrorHTTPStatusMessage=Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature. ErrorHTTPHeaders={"x-ms-error-code":"AuthorizationFailure","x-ms-request-id":"6c4b9643-4050-4b51-84de-ba11e1636e36"} ErrorHTTPBody="<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>AuthorizationFailure</Code>\n  <Message>Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.\nRequestId:6c4b9643-4050-4b51-84de-ba11e1636e36\nTime:2022-01-05T12:21:14.389Z</Message>\n</Error>" ErrorStack="StorageError: Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.\n    at Function.getAuthorizationFailure (/opt/azurite/dist/src/blob/errors/StorageErrorFactory.js:134:16)\n    at /opt/azurite/dist/src/blob/middlewares/AuthenticationMiddlewareFactory.js:22:56\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)"
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Set HTTP code: 403
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Set HTTP status message: Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Set HTTP Header: x-ms-error-code=AuthorizationFailure
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Set HTTP Header: x-ms-request-id=6c4b9643-4050-4b51-84de-ba11e1636e36
2022-01-05T12:21:14.390Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Set content type: application/xml
2022-01-05T12:21:14.391Z 6c4b9643-4050-4b51-84de-ba11e1636e36 error: ErrorMiddleware: Set HTTP body: "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>AuthorizationFailure</Code>\n  <Message>Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.\nRequestId:6c4b9643-4050-4b51-84de-ba11e1636e36\nTime:2022-01-05T12:21:14.389Z</Message>\n</Error>"
2022-01-05T12:21:14.391Z 6c4b9643-4050-4b51-84de-ba11e1636e36 info: EndMiddleware: End response. TotalTimeInMS=5 StatusCode=403 StatusMessage=Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature. Headers={"server":"Azurite-Blob/3.15.0","x-ms-error-code":"AuthorizationFailure","x-ms-request-id":"6c4b9643-4050-4b51-84de-ba11e1636e36","content-type":"application/xml"}
2022-01-05T12:21:14.597Z 5b97d741-b8b3-47d2-ae55-752b3f669f1f info: QueueStorageContextMiddleware: RequestMethod=GET RequestURL=http://azurite/devstoreaccount1/azure-webjobs-blobtrigger-func-610388490/messages?numofmessages=16&visibilitytimeout=600 RequestHeaders:{"host":"azurite:10001","x-ms-client-request-id":"6b565ba5-fd88-4f47-a6db-c29dc0d57385","user-agent":"Azure-Storage/11.1.7 (.NET Core; Unix 5.15.10.200)","x-ms-version":"2019-02-02","x-ms-date":"Wed, 05 Jan 2022 12:21:14 GMT","authorization":"SharedKey devstoreaccount1:iZbBKP31qfKXJlxVGOZwGcZAjgxJNWPpAFeCFSm5Kw8="} ClientIP=172.24.0.9 Protocol=http HTTPVersion=1.1
2022-01-05T12:21:14.598Z 5b97d741-b8b3-47d2-ae55-752b3f669f1f info: QueueStorageContextMiddleware: Account=devstoreaccount1 Queue=azure-webjobs-blobtrigger-func-610388490 Message=messages MessageId=undefined
2022-01-05T12:21:14.598Z 5b97d741-b8b3-47d2-ae55-752b3f669f1f verbose: DispatchMiddleware: Dispatching request...
2022-01-05T12:21:14.598Z 5b97d741-b8b3-47d2-ae55-752b3f669f1f info: DispatchMiddleware: Operation=Messages_Dequeue
2022-01-05T12:21:14.598Z 5b97d741-b8b3-47d2-ae55-752b3f669f1f verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2022-01-05T12:21:14.598Z 5b97d741-b8b3-47d2-ae55-752b3f669f1f info: QueueSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
/opt/azurite #
hholst80 commented 2 years ago

I finally found a workaround to this. I would still very much like to understand why the SAS key generated by the Azure Function is not valid.

The workaround is to put the function container in the same container as azurite and then use localhost in local.settings.json

  func:
    network_mode: service:azurite
    container_name: func
#   hostname: func
    build: func
    # ....
  azurite:
    container_name: azurite
    hostname: azurite
    image: mcr.microsoft.com/azure-storage/azurite:latest
    command: azurite --blobHost 0.0.0.0 --queueHost 0.0.0.0 --tableHost 0.0.0.0 -d /tmp/debug.log
    ports:
      - 10000-10002:10000-10002
    working_dir: /azurite
    volumes:
      - azurite:/azurite

{
  "IsEncrypted": false,
  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;QueueEndpoint=http://localhost:10001/devstoreaccount1;TableEndpoint=http://localhost:10002/devstoreaccount1;"
  }
}
blueww commented 2 years ago

@hholst80 Thanks for the update! Good to know you find a way to workaround that. This looks like related with some azure function config. We will need time to look deep into it.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

bpossolo commented 2 years ago

try upgrading to azurite 3.17.1 there were several bugs related to SAS validation in azurite versions < 3.17.0

see here for some examples:

hholst80 commented 2 years ago

I will look into it this weekend hopefully.

piotr-napadlek commented 7 months ago

I'm still experiencing the same issue; SAS token generated with java sdk:

    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-blob</artifactId>
      <version>12.25.2</version>  <!-- {x-version-update;com.azure:azure-storage-blob;dependency} -->
    </dependency>

azurite version:

mcr.microsoft.com/azure-storage/azurite:3.29.0

Sample URL generated by following code

        BlobClient blobClient = blobContainerClient.getBlobClient(blobPath);
        return blobClient.getBlobUrl() + "&" + blobClient.generateSas(new BlobServiceSasSignatureValues(OffsetDateTime.now(ZoneOffset.UTC).plusDays(3),
                new BlobSasPermission().setReadPermission(true)).setProtocol(SasProtocol.HTTPS_HTTP));

Sample generated url:

http://localhost:10000/devstoreaccount1/<redacted>/<redacted>.json.gz&sv=2023-11-03&spr=https%2Chttp&se=2024-03-16T13%3A43%3A57Z&sr=b&sp=r&sig=Yh5%2BsgAHLCR2mOn%2FQp4Mkuh1IFqGU%2BFsVYUKBEHc4i4%3D

Response from the url:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>AuthorizationFailure</Code>
    <Message>Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.
RequestId:36d3eda5-aa94-46ab-8f35-f1c03d78feb0
Time:2024-03-13T13:44:15.382Z</Message>
</Error>

Tried both localhost and 127.0.01 in the connection string

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;QueueEndpoint=http://localhost:10001/devstoreaccount1;TableEndpoint=http://localhost:10002/devstoreaccount1;

as well as simple

UseDevelopmentStorage=true;

all with the same result