search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.8k stars 320 forks source link

azure-sdk-for-go AppendBlock operation resolves to AppendBlobHandler.create() #702

Open mihaitodor opened 3 years ago

mihaitodor commented 3 years ago

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

Latest (v3.11.0)

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub (mcr.microsoft.com/azure-storage/azurite:3.11.0)

What's the Node.js version?

Whatever the mcr.microsoft.com/azure-storage/azurite:3.11.0 docker container is running

What problem was encountered?

Blob.AppendBlock gets resolved to AppendBlobHandler.create() instead of AppendBlobHandler.appendBlock(). This is happening because the API specification requires the comp query parameter for the AppendBlob_AppendBlock operation, while the AppendBlob_Create operation requires the x-ms-blob-type header, which makes it equivalent to the AppendBlob_Create operation in dispatchMiddleware (keys 59 and 60 return the same conditionsMet) because both of them contain the Content-Length header.

Would it be possible to make the x-ms-blob-type header mandatory for the AppendBlob_AppendBlock operation? This will probably require doing the same for AppendBlob_AppendBlockFromUrl, so it won't end up being confused with AppendBlob_AppendBlock. I'm happy to submit a PR for this if you can provide the steps to re-generate the code from the swagger definition.

Steps to reproduce the issue?

Run the following code:

package main

import (
    "io/ioutil"
    "log"
    "time"

    "github.com/Azure/azure-sdk-for-go/storage"
    "github.com/google/uuid"
    "github.com/ory/dockertest/v3"
    "github.com/ory/dockertest/v3/docker"
)

// RunAzurite starts an azurite container
func RunAzurite(pool *dockertest.Pool) (*dockertest.Resource, error) {
    opts := dockertest.RunOptions{
        Repository: "mcr.microsoft.com/azure-storage/azurite",
        Tag:        "3.11.0",
        PortBindings: map[docker.Port][]docker.PortBinding{
            "10000/tcp": {{HostIP: "0.0.0.0", HostPort: "10000"}},
        },
        Cmd: []string{"azurite", "--blobHost", "0.0.0.0", "--loose"},
    }
    azurite, err := pool.RunWithOptions(&opts)
    if err != nil {
        return nil, err
    }
    if eerr := azurite.Expire(10); eerr != nil {
        return nil, eerr
    }
    pool.MaxWait = 10 * time.Second
    rerr := pool.Retry(func() error {
        client, eerr := storage.NewEmulatorClient()
        if eerr != nil {
            return eerr
        }
        s := client.GetBlobService()
        c := s.GetContainerReference("cont")
        if _, err = c.Exists(); err != nil {
            return err
        }
        return nil
    })
    return azurite, rerr
}

func main() {
    pool, err := dockertest.NewPool("")
    if err != nil {
        log.Fatalf("Failed to create dockertest pool: %s", err)
    }

    azurite, err := RunAzurite(pool)
    defer pool.Purge(azurite)

    if err != nil {
        log.Fatalf("Failed to start Azurite: %s", err)
    }

    client, err := storage.NewEmulatorClient()
    if err != nil {
        log.Fatalf("Failed to create storage client: %s", err)
    }

    blobClient := client.GetBlobService()
    dummyContainer := uuid.New().String()
    containerRef := blobClient.GetContainerReference(dummyContainer)

    err = containerRef.Create(nil)
    if err != nil {
        log.Fatalf("Failed to create container: %s", err)
    }

    dummyBlob := uuid.New().String()
    blobRef := containerRef.GetBlobReference(dummyBlob)

    err = blobRef.PutAppendBlob(nil)
    if err != nil {
        log.Fatalf("Failed to create append blob: %s", err)
    }

    dummyData := uuid.New().String()
    err = blobRef.AppendBlock([]byte(dummyData), nil)
    if err != nil {
        log.Fatalf("Failed to write to append blob: %s", err)
    }

    blob, err := blobRef.Get(nil)
    if err != nil {
        log.Fatalf("Failed to get blob: %s", err)
    }

    data, err := ioutil.ReadAll(blob)
    if err != nil {
        log.Fatalf("Failed to read blob: %s", err)
    }

    if string(data) != dummyData {
        log.Fatalf("Expected %q but received %q instead", dummyData, string(data))
    }

    log.Print("That's all folks!")
}

Have you found a mitigation/solution?

Sadly, no workaround...

Would it be possible to make the x-ms-blob-type header mandatory for the AppendBlob_AppendBlock operation? This will probably require doing the same for AppendBlob_AppendBlockFromUrl, so it won't end up being confused with AppendBlob_AppendBlock. I'm happy to submit a PR for this if you can provide the steps to re-generate the code from the swagger definition.

I would very much hope to see this fixed, if possible. Upgrading to the new azure-storage-blob-go API is not feasible for now (although I think that one might encounter the same issue).

mihaitodor commented 3 years ago

Small update: I tried running autorest --typescript --output-folder=./../src/blob/generated --clear-output-folder --input-file=blob-storage-2019-02-02.json from the swagger folder and got the following error:

AutoRest code generation utility [cli version: 3.0.6338; node: v10.23.3, max-memory: 1280 MB]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
NOTE: AutoRest core version selected from configuration: ~3.0.6298.
   Loading AutoRest core      '/root/.autorest/@autorest_core@3.0.6372/node_modules/@autorest/core/dist' (3.0.6372)
   Loading AutoRest extension '@autorest/typescript' (6.0.0-dev.20201105.2->6.0.0-dev.20201105.2)
   Loading AutoRest extension '@autorest/modelerfour' (4.15.421->4.15.421)

WARNING (PreCheck/SchemaMissingType): The schema 'DataLakeStorageError-error' with an undefined type and decalared properties is a bit ambigious. This has been auto-corrected to 'type:object'

WARNING (PreCheck/SchemaMissingType): The schema 'Metrics' with an undefined type and decalared properties is a bit ambigious. This has been auto-corrected to 'type:object'

WARNING (PreCheck/CheckDuplicateSchemas): Checking for duplicate schemas, this could take a (long) while.  Run with --verbose for more detail.

ERROR (): Duplicate object schemas with 'DataLakeStorageError' name  detected.
FATAL: Error: 1 errors occured -- cannot continue.

I don't see any duplication of the DataLakeStorageError schema in blob-storage-2019-02-02.json, but maybe I'm doing something wrong.

blueww commented 3 years ago

@mihaitodor

Blob.AppendBlock gets resolved to AppendBlobHandler.create() instead of AppendBlobHandler.appendBlock().

[Wei] I am a little confused on this. Do you mean you would like to append block, but azurite actually created append blob? Per the rest doc of create blob and append block, you need add "?comp=appendblock" in the Uri to indicate you need append block. Have you add it when try to append block? Would you please share the Azurite debug log of the request with issue, and we will be more clear on this.

Would it be possible to make the x-ms-blob-type header mandatory for the AppendBlob_AppendBlock operation?

[Wei] per the rest doc of append block, it does not support x-ms-blob-type header. Why you want to add it, or even make it mandatory? This might will break other user of Azurite. Besides that, x-ms-blob-type is mandatory for create blob rest api.

mihaitodor commented 3 years ago

Hey @blueww, thank you for looking into this issue and sharing links to the relevant documentation!

Here is the Azurite debug log (for v3.11.0)

2021-02-26T21:47:22.333Z info: Azurite Blob service is starting on 0.0.0.0:10000 2021-02-26T21:47:22.334Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined 2021-02-26T21:47:22.334Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1. 2021-02-26T21:47:22.347Z info: BlobGCManager:start() Starting BlobGCManager. Set status to Initializing. 2021-02-26T21:47:22.347Z info: BlobGCManager:start() Trigger mark and sweep loop. Set status to Running. 2021-02-26T21:47:22.347Z info: BlobGCManager:markSweepLoop() Start next mark and sweep. 2021-02-26T21:47:22.347Z info: BlobGCManager:markSweep() Get all extents. 2021-02-26T21:47:22.348Z info: BlobGCManager:start() BlobGCManager successfully started. 2021-02-26T21:47:22.349Z info: BlobGCManager:markSweep() Got 0 extents. 2021-02-26T21:47:22.350Z info: BlobGCManager:markSweep() Get referred extents. 2021-02-26T21:47:22.350Z info: BlobGCManager:markSweep() Got referred extents, unreferenced extents count is 0. 2021-02-26T21:47:22.350Z info: BlobGCManager:markSweepLoop() Mark and sweep finished, taken 3ms. 2021-02-26T21:47:22.350Z info: BlobGCManager:markSweepLoop() Sleep for 600000ms. 2021-02-26T21:47:22.352Z info: Azurite Blob service successfully listens on http://0.0.0.0:10000 2021-02-26T21:47:22.352Z info: Azurite Queue service is starting on 127.0.0.1:10001 2021-02-26T21:47:22.352Z info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined 2021-02-26T21:47:22.352Z info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1. 2021-02-26T21:47:22.359Z info: QueueGCManager:start() Starting QueueGCManager, set status to Initializing 2021-02-26T21:47:22.359Z info: QueueGCManager:start() Trigger mark and sweep loop, set status to Running. 2021-02-26T21:47:22.359Z info: QueueGCManager:markSweepLoop() Start new mark and sweep. 2021-02-26T21:47:22.359Z info: QueueGCManger:markSweep() Get all extents. 2021-02-26T21:47:22.359Z info: QueueGCManager:start() QueueGCManager successfully started. 2021-02-26T21:47:22.360Z info: QueueGCManager:marksweep() Get 0 extents. 2021-02-26T21:47:22.360Z info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents. 2021-02-26T21:47:22.360Z info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0. 2021-02-26T21:47:22.360Z info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms. 2021-02-26T21:47:22.360Z info: QueueGCManager:markSweepLoop() Sleep for 60000 2021-02-26T21:47:22.360Z info: Azurite Queue service successfully listens on http://127.0.0.1:10001 2021-02-26T21:47:27.260Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152?restype=container RequestHeaders:{"host":"127.0.0.1:10000","user-agent":"Go/go1.15.6 (amd64-darwin) azure-storage-go/v52.0.0 api-version/2018-03-28 blob","content-length":"0","authorization":"SharedKey devstoreaccount1:P5vpn9S1tsXYMQ/z/Ri0zA+JOv3088KQ7dzBidXZiCE=","x-ms-date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-version":"2018-03-28","accept-encoding":"gzip"} ClientIP=172.17.0.1 Protocol=http HTTPVersion=1.1 2021-02-26T21:47:27.260Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 Blob= 2021-02-26T21:47:27.261Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 verbose: DispatchMiddleware: Dispatching request... 2021-02-26T21:47:27.263Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: DispatchMiddleware: Operation=Container_Create 2021-02-26T21:47:27.264Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications. 2021-02-26T21:47:27.264Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: PublicAccessAuthenticator:validate() Start validation against public access. 2021-02-26T21:47:27.264Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 debug: PublicAccessAuthenticator:validate() Getting account properties... 2021-02-26T21:47:27.264Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152, blob: 2021-02-26T21:47:27.270Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 2021-02-26T21:47:27.271Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication. 2021-02-26T21:47:27.276Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Fri, 26 Feb 2021 21:47:27 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152\nrestype:container" 2021-02-26T21:47:27.277Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:P5vpn9S1tsXYMQ/z/Ri0zA+JOv3088KQ7dzBidXZiCE= 2021-02-26T21:47:27.277Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched. 2021-02-26T21:47:27.277Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 verbose: DeserializerMiddleware: Start deserializing... 2021-02-26T21:47:27.278Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{}},"restype":"container","version":"2018-03-28"} 2021-02-26T21:47:27.279Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 verbose: SerializerMiddleware: Start serializing... 2021-02-26T21:47:27.280Z c40f3469-b6bb-4517-bfda-7a4651c6efd9 info: EndMiddleware: End response. TotalTimeInMS=20 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.11.0","etag":"\"0x20D9098E314E120\"","last-modified":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-request-id":"c40f3469-b6bb-4517-bfda-7a4651c6efd9","x-ms-version":"2020-06-12"} 2021-02-26T21:47:27.287Z bb36c182-099c-46aa-841d-55a279c9495d info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152/9ae8f571-dce6-4163-b627-ad42b3302dd2 RequestHeaders:{"host":"127.0.0.1:10000","user-agent":"Go/go1.15.6 (amd64-darwin) azure-storage-go/v52.0.0 api-version/2018-03-28 blob","content-length":"0","authorization":"SharedKey devstoreaccount1:in/gPKc+NKO7eqhIk/wbvwxvxG2xjGEpI9YJf3B7k9Y=","x-ms-blob-type":"AppendBlob","x-ms-date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-version":"2018-03-28","accept-encoding":"gzip"} ClientIP=172.17.0.1 Protocol=http HTTPVersion=1.1 2021-02-26T21:47:27.287Z bb36c182-099c-46aa-841d-55a279c9495d info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 Blob=9ae8f571-dce6-4163-b627-ad42b3302dd2 2021-02-26T21:47:27.287Z bb36c182-099c-46aa-841d-55a279c9495d verbose: DispatchMiddleware: Dispatching request... 2021-02-26T21:47:27.289Z bb36c182-099c-46aa-841d-55a279c9495d info: DispatchMiddleware: Operation=AppendBlob_Create 2021-02-26T21:47:27.289Z bb36c182-099c-46aa-841d-55a279c9495d verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications. 2021-02-26T21:47:27.289Z bb36c182-099c-46aa-841d-55a279c9495d info: PublicAccessAuthenticator:validate() Start validation against public access. 2021-02-26T21:47:27.289Z bb36c182-099c-46aa-841d-55a279c9495d debug: PublicAccessAuthenticator:validate() Getting account properties... 2021-02-26T21:47:27.289Z bb36c182-099c-46aa-841d-55a279c9495d debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152, blob: 9ae8f571-dce6-4163-b627-ad42b3302dd2 2021-02-26T21:47:27.291Z bb36c182-099c-46aa-841d-55a279c9495d debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 2021-02-26T21:47:27.291Z bb36c182-099c-46aa-841d-55a279c9495d info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication. 2021-02-26T21:47:27.291Z bb36c182-099c-46aa-841d-55a279c9495d info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-blob-type:AppendBlob\nx-ms-date:Fri, 26 Feb 2021 21:47:27 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152/9ae8f571-dce6-4163-b627-ad42b3302dd2" 2021-02-26T21:47:27.291Z bb36c182-099c-46aa-841d-55a279c9495d info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:in/gPKc+NKO7eqhIk/wbvwxvxG2xjGEpI9YJf3B7k9Y= 2021-02-26T21:47:27.291Z bb36c182-099c-46aa-841d-55a279c9495d info: BlobSharedKeyAuthenticator:validate() Signature 1 matched. 2021-02-26T21:47:27.291Z bb36c182-099c-46aa-841d-55a279c9495d verbose: DeserializerMiddleware: Start deserializing... 2021-02-26T21:47:27.292Z bb36c182-099c-46aa-841d-55a279c9495d info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"blobHTTPHeaders":{},"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"contentLength":0,"version":"2018-03-28","blobType":"AppendBlob"} 2021-02-26T21:47:27.292Z bb36c182-099c-46aa-841d-55a279c9495d verbose: SerializerMiddleware: Start serializing... 2021-02-26T21:47:27.293Z bb36c182-099c-46aa-841d-55a279c9495d info: EndMiddleware: End response. TotalTimeInMS=6 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.11.0","etag":"\"0x1C692651BAB9E60\"","last-modified":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-request-id":"bb36c182-099c-46aa-841d-55a279c9495d","x-ms-version":"2020-06-12","date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-request-server-encrypted":"true"} 2021-02-26T21:47:27.295Z 380f8116-8589-4751-854f-84b5f356aa4e info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://127.0.0.1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152/9ae8f571-dce6-4163-b627-ad42b3302dd2?comp=appendblock RequestHeaders:{"host":"127.0.0.1:10000","user-agent":"Go/go1.15.6 (amd64-darwin) azure-storage-go/v52.0.0 api-version/2018-03-28 blob","content-length":"36","authorization":"SharedKey devstoreaccount1:zCFgKfXAuGAnAL/BrOTS+5rqGTK+rNF6TKuN1l5CB2g=","x-ms-blob-type":"AppendBlob","x-ms-date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-version":"2018-03-28","accept-encoding":"gzip"} ClientIP=172.17.0.1 Protocol=http HTTPVersion=1.1 2021-02-26T21:47:27.295Z 380f8116-8589-4751-854f-84b5f356aa4e info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 Blob=9ae8f571-dce6-4163-b627-ad42b3302dd2 2021-02-26T21:47:27.295Z 380f8116-8589-4751-854f-84b5f356aa4e verbose: DispatchMiddleware: Dispatching request... 2021-02-26T21:47:27.296Z 380f8116-8589-4751-854f-84b5f356aa4e info: DispatchMiddleware: Operation=AppendBlob_Create 2021-02-26T21:47:27.296Z 380f8116-8589-4751-854f-84b5f356aa4e verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications. 2021-02-26T21:47:27.296Z 380f8116-8589-4751-854f-84b5f356aa4e info: PublicAccessAuthenticator:validate() Start validation against public access. 2021-02-26T21:47:27.296Z 380f8116-8589-4751-854f-84b5f356aa4e debug: PublicAccessAuthenticator:validate() Getting account properties... 2021-02-26T21:47:27.296Z 380f8116-8589-4751-854f-84b5f356aa4e debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152, blob: 9ae8f571-dce6-4163-b627-ad42b3302dd2 2021-02-26T21:47:27.297Z 380f8116-8589-4751-854f-84b5f356aa4e debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 2021-02-26T21:47:27.297Z 380f8116-8589-4751-854f-84b5f356aa4e info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication. 2021-02-26T21:47:27.297Z 380f8116-8589-4751-854f-84b5f356aa4e info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n36\n\n\n\n\n\n\n\n\nx-ms-blob-type:AppendBlob\nx-ms-date:Fri, 26 Feb 2021 21:47:27 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152/9ae8f571-dce6-4163-b627-ad42b3302dd2\ncomp:appendblock" 2021-02-26T21:47:27.297Z 380f8116-8589-4751-854f-84b5f356aa4e info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:zCFgKfXAuGAnAL/BrOTS+5rqGTK+rNF6TKuN1l5CB2g= 2021-02-26T21:47:27.297Z 380f8116-8589-4751-854f-84b5f356aa4e info: BlobSharedKeyAuthenticator:validate() Signature 1 matched. 2021-02-26T21:47:27.297Z 380f8116-8589-4751-854f-84b5f356aa4e verbose: DeserializerMiddleware: Start deserializing... 2021-02-26T21:47:27.298Z 380f8116-8589-4751-854f-84b5f356aa4e info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"blobHTTPHeaders":{},"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"contentLength":36,"version":"2018-03-28","blobType":"AppendBlob"} 2021-02-26T21:47:27.299Z 380f8116-8589-4751-854f-84b5f356aa4e verbose: SerializerMiddleware: Start serializing... 2021-02-26T21:47:27.299Z 380f8116-8589-4751-854f-84b5f356aa4e info: EndMiddleware: End response. TotalTimeInMS=4 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.11.0","etag":"\"0x19F6A8A31A42460\"","last-modified":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-request-id":"380f8116-8589-4751-854f-84b5f356aa4e","x-ms-version":"2020-06-12","date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-request-server-encrypted":"true"} 2021-02-26T21:47:27.302Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152/9ae8f571-dce6-4163-b627-ad42b3302dd2 RequestHeaders:{"host":"127.0.0.1:10000","user-agent":"Go/go1.15.6 (amd64-darwin) azure-storage-go/v52.0.0 api-version/2018-03-28 blob","authorization":"SharedKey devstoreaccount1:2WBxQmmDNT32OUjmK6mTs7tQ+idFWXlNBek5B1jGB7I=","x-ms-date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-version":"2018-03-28","accept-encoding":"gzip"} ClientIP=172.17.0.1 Protocol=http HTTPVersion=1.1 2021-02-26T21:47:27.302Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 Blob=9ae8f571-dce6-4163-b627-ad42b3302dd2 2021-02-26T21:47:27.302Z fb304864-fb19-4635-9904-a4a43f98ad64 verbose: DispatchMiddleware: Dispatching request... 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 info: DispatchMiddleware: Operation=Blob_Download 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications. 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 info: PublicAccessAuthenticator:validate() Start validation against public access. 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 debug: PublicAccessAuthenticator:validate() Getting account properties... 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152, blob: 9ae8f571-dce6-4163-b627-ad42b3302dd2 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container 21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication. 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Fri, 26 Feb 2021 21:47:27 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/21d5acf1-ae1e-4b9b-a0dd-5fcb34b68152/9ae8f571-dce6-4163-b627-ad42b3302dd2" 2021-02-26T21:47:27.303Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:2WBxQmmDNT32OUjmK6mTs7tQ+idFWXlNBek5B1jGB7I= 2021-02-26T21:47:27.304Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched. 2021-02-26T21:47:27.304Z fb304864-fb19-4635-9904-a4a43f98ad64 verbose: DeserializerMiddleware: Start deserializing... 2021-02-26T21:47:27.304Z fb304864-fb19-4635-9904-a4a43f98ad64 info: HandlerMiddleware: DeserializedParameters={"options":{"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"version":"2018-03-28"} 2021-02-26T21:47:27.305Z fb304864-fb19-4635-9904-a4a43f98ad64 info: BlobHandler:downloadBlockBlobOrAppendBlob() NormalizedDownloadRange=bytes=0--1 RequiredContentLength=0 2021-02-26T21:47:27.306Z fb304864-fb19-4635-9904-a4a43f98ad64 verbose: SerializerMiddleware: Start serializing... 2021-02-26T21:47:27.307Z fb304864-fb19-4635-9904-a4a43f98ad64 info: Serializer: Start returning stream body. 2021-02-26T21:47:27.308Z fb304864-fb19-4635-9904-a4a43f98ad64 info: EndMiddleware: End response. TotalTimeInMS=6 StatusCode=200 StatusMessage=OK Headers={"server":"Azurite-Blob/3.11.0","last-modified":"Fri, 26 Feb 2021 21:47:27 GMT","content-length":"0","content-type":"application/octet-stream","etag":"\"0x19F6A8A31A42460\"","content-md5":"1B2M2Y8AsgTpgAmY7PhCfg==","x-ms-blob-type":"AppendBlob","x-ms-lease-state":"available","x-ms-lease-status":"unlocked","x-ms-request-id":"fb304864-fb19-4635-9904-a4a43f98ad64","x-ms-version":"2020-06-12","accept-ranges":"bytes","date":"Fri, 26 Feb 2021 21:47:27 GMT","x-ms-blob-committed-block-count":"0","x-ms-server-encrypted":"true"} 2021-02-26T21:47:37.208Z info: Azurite Blob service is closing... 2021-02-26T21:47:37.208Z info: Azurite Queue service is closing... 2021-02-26T21:47:37.209Z info: BlobGCManager:close() Start closing BlobGCManager. Set status to Closing. 2021-02-26T21:47:37.209Z info: QueueGCManager:close() Start closing QueueGCManager, set status to Closing. 2021-02-26T21:47:37.209Z info: BlobGCManager:start() Mark and sweep loop is closed. 2021-02-26T21:47:37.210Z info: BlobGCManager:close() BlobGCManager successfully closed. Set status to Closed. 2021-02-26T21:47:37.210Z info: QueueGCManager:start() Mark and sweep loop is closed. 2021-02-26T21:47:37.210Z info: QueueGCManager:close() QueueGCManager successfully closed, set status to Closed. 2021-02-26T21:47:37.211Z info: Azurite Blob service successfully closed 2021-02-26T21:47:37.211Z info: Azurite Queue service successfully closed

In there, you can observe that DispatchMiddleware: Operation=AppendBlob_Create is selected twice, although the second request URL ends with ?comp=appendblock, so it never actually writes the content when blobRef.AppendBlock is called in the code.

per the rest doc of append block, it does not support x-ms-blob-type header. Why you want to add it, or even make it mandatory? This might will break other user of Azurite. Besides that, x-ms-blob-type is mandatory for create blob rest api.

That's understandable, but I feel this REST API has been formalised after azure-sdk-for-go was developed and they're not fully compatible. However, anyone who would like to test their existing code using Azurite won't be able to call AppendBlock, as my code sample demonstrates.

My proposal to mark the x-ms-blob-type header as mandatory for AppendBlock (and AppendBlockFromUrl) in Azurite is my naive attempt to getting Azurite to support this functionality when invoked from azure-sdk-for-go, since I understand that library is unlikely to be changed given its legacy status. Unfortunately, I can't switch to the new storage SDK just yet as mentioned before.

I assumed that the lack of the x-ms-blob-type header from the AppendBlock and AppendBlockFromUrl API calls is an omission, but maybe I'm not seeing some pattern. However, given that it's documented this way, I do agree that mandating it might break other client libraries, which is undesirable.

As an alternative, I put together a hack in #710, which is only active when the --loose flag is set. If you think this is acceptable, it seems to do the job and shouldn't impact any other client libraries. Please let me know if you'd prefer to follow a different approach.

blueww commented 3 years ago

@mihaitodor As this is Go legacy SDK issue, it might not be so good to change Azurite for it. Would you like try to raise a PR for Go SDK repo to fix the issue? This might be a better way to fix the issue.

mihaitodor commented 3 years ago

@blueww Unfortunately, that code is quite tangled and brittle. I fear changing this behaviour in it might cause unintended side-effects.

blueww commented 3 years ago

@mihaitodor

Would you please file an issue on Go SDK repo https://github.com/Azure/azure-sdk-for-go, and see if Go SDK owner can help to fix it.

Besides that, I will also try to ping the SDK owner to see if they can help to fix it.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.