search

Azure / Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
MIT License
1.83k stars 325 forks source link

blob trigger function not working with azurite #246

Open bigpod98 opened 5 years ago

bigpod98 commented 5 years ago

i Just tried running blob trigger function that uses azurite as storage coppied connection string from storage explorer im probably messing up. to make function i use c#, and to host azurite i use docker container based azurite

all im getting error is that storage account isnt configured

XiaoningLiu commented 5 years ago

@bigpod98 I don't quite understand. Can you provide a reproduce steps? The error message "storage account isnt configured" doesn't sound like returned by Azurite

bigpod98 commented 5 years ago

I Just made a c# based funtion in visual studio added in the connection String ( DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint= http://192.168.100.3:10000/devstoreaccount1; ) then i run the function And i get thats error.

I use docker to host azurite I use Ubuntu to host docker

I added the String to local.settings.json And the function

My guess im might be doing something wrong

V pon., 5. avg. 2019 04:59 je oseba xiaonlimsft notifications@github.com napisala:

@bigpod98 https://github.com/bigpod98 I don't quite understand. Can you provide a reproduce steps? The error message "storage account isnt configured" doesn't sound like returned by Azurite

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Azure/Azurite/issues/246?email_source=notifications&email_token=ABUZ5OKLWTAPN5F23GPVU7DQC6JQHA5CNFSM4IJEER62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QRP5A#issuecomment-518068212, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUZ5OP4JU2PVBC77DYV7NTQC6JQHANCNFSM4IJEER6Q .

XiaoningLiu commented 5 years ago

@bigpod98 I'm guessing whether the Azurite instance has been successfully started or not. Can you try to connect with Azurite using Azure Storage Explorer? To make sure Azurite has been setup properly.

And by default, Azurite will output access logs to console for incoming requests.

bigpod98 commented 5 years ago

@XiaoningLiu sorry for so late response i can connect to the azurite using storage explorer and i can upload files using c# program i wrote

XiaoningLiu commented 5 years ago

Glad to hear about this. Close this issue.

bigpod98 commented 5 years ago

@XiaoningLiu I think you missunderstood i can connect trought another c# program and upload but it doesnt work when using as blob trigger which is original problem

bigpod98 commented 5 years ago

@XiaoningLiu i think i should explain because now that i read it ia quite confusing. I have 2 c# based programs one is a function with blob trigger another is client program that uploads files to container. Files uploaded to container then trigger blob trigger but blob trigger function outputs error mentioned above. I opened function in function core tools and as a docker container both output same/simular error but i mostly reffer to the error from core tools

XiaoningLiu commented 5 years ago

Hi, I don't think it's Azurite issue because it works for Storage Explorer and client C# program. I'm confused about the funtion blob trigger. It sounds like the blob trigger is kind of cloud based deployment, and I don't know how does it work.

bigpod98 commented 5 years ago

thanks for information my guess what something like that as well i know it just says that it says that storage account is not set when i put in the key and the function core tools are ment to be used to test on your own hardware and with your own storage it does work with Storage Emulator and as far as i know azurite works with blob trigger functions if it is set to be developer storage

bigpod98 commented 5 years ago

@XiaoningLiu Let me try to provide more my local.settings.json { "IsEncrypted": false, "Values": { "AzureWebJobsStorage": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://192.168.100.3:10000/devstoreaccount1;", "FUNCTIONS_WORKER_RUNTIME": "dotnet" } }

this is one function: public static class Function { [FunctionName("function")] public static void Run([BlobTrigger("container/{name}", Connection = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://192.168.100.3:10000/devstoreaccount1;")]Stream myBlob, string name, ILogger log) { log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes"); } }

bigpod98 commented 5 years ago

@XiaoningLiu Microsoft.Azure.WebJobs.Host: Error indexing method 'function'. Microsoft.Azure.WebJobs.Extensions.Storage: Storage account 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://192.168.100.3:10000/devstoreaccount1;' is not configured.

this is the error

i also just saw a warning before which might explain things Cannot find value named 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://192.168.100.3:10000/devstoreaccount1;' in local.settings.json that matches 'connection' property set on 'blobTrigger' in 'Location'. You can run 'func azure functionapp fetch-app-settings <functionAppName>' or specify a connection string in local.settings.json.

XiaoningLiu commented 5 years ago

@bigpod98 Does Azurite reports any errors during above process? It looks like blobTrigger doesn't have good configuration just like your warning logs.

bigpod98 commented 5 years ago

@XiaoningLiu as far as i are there is no reaction from azurite i might be missing log files but Just watching output od azurite is see when i put in container file but when opening function nothing happens

mydiemho commented 5 years ago

@bigpod98 replace the value for AzureWebJobsStorage with 

    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
  1. How are you running your function app? Is it also inside the same docker container you're running Azurite?
  2. If you're running your function app locally but running azurite inside Docker, you have to make sure that the same port on your host is mapped to the port in docker where Azurite is running, ie. 10000
AdamUnderwood commented 5 years ago

@mydiemho -- I'm hoping to get some answers on this as well. I've got a basic functions app running in Docker-compose with azurite + azure-functions/python:2.0. It's very simple -- 1 blob trigger, 1 queue trigger. The blob trigger adds to the queue and the queue trigger is fired.

If I add a file to the container, nothing happens. I see the logs (storage_1 being the container name): storage_1 | 192.168.xxx.x - - [11/Sep/2019:15:32:21 +0000] "GET /devstoreaccount1/testing?restype=container&comp=list HTTP/1.1" 200 - storage_1 | 192.168.xxx.x - - [11/Sep/2019:15:34:12 +0000] "PUT /devstoreaccount1/azure-webjobs-hosts/locks/6d0caf34af9a-2137340777/host?comp=lease HTTP/1.1" 200 -

If I add to the queue directly in storage explorer, the queue trigger is fired. I'm wondering why it isn't picking up blob additions. This works when set to a container in azure. I don't know if this is an azurite problem or a problem with azure-functions, but it's strange it works with local queues, but not local blobs.

My docker compose file:

version: '3.1'
services:
  db:
    image: postgres
    ports:
      - '5432:5432'
    volumes:
      - pgdata:/var/lib/postgresql/data
    env_file:
      - local.env

  functions:
    build:
      context: ./azure-functions/
    env_file:
      - local.env
    volumes:
      - ./azure-functions/shared:/home/site/wwwroot/shared
    depends_on:
      - storage

  storage:
    image: mcr.microsoft.com/azure-storage/azurite
    ports: 
      - "10000:10000" 
      - "10001:10001"

volumes:
  pgdata:

And local.env:

AzureWebJobsStorage=UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://storage;
AzureStorageQueuesConnectionString=UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://storage;

Am I missing something obvious?

bigpod98 commented 5 years ago

@mydiemho azurite is tunning in docker container while function is running on same server directly and correct ports are published(same ports on inside and ousode the container)

mydiemho commented 5 years ago

@bigpod98 did you update the value for AzureWebJobsStorage and retry?

mydiemho commented 5 years ago

@AdamUnderwood I have not try running this in container.

  1. is the azurite image an official image? Can you double check what version it's running?
  2. I'm wondering why it isn't picking up blob additions. Have you verify that your blob-trigger function is writing to the queue?
  3. I can't say for sure if it's function or azurite related until I can see your function code. Can you post sample of your handler code?
bigpod98 commented 5 years ago

@mydiemho not yet, had to send my dev machine to service center and had to dust out the old one

AdamUnderwood commented 5 years ago

@mydiemho thanks for the reply: I'm using mcr.microsoft.com/azure-storage/azurite as the image.
Function.json:

    "scriptFile": "__init__.py",
    "bindings": [
        {
            "name": "myblob",
            "type": "blobTrigger",
            "direction": "in",
            "path": "testing/{name}",
            "connection":"AzureWebJobsStorage"
        }
    ]
}

init.py:

import logging
import os
import azure.functions as func
from azure.storage.queue import QueueService, QueueMessageFormat

def main(myblob: func.InputStream):
    logging.info(f"Python blob trigger function processed blob \n"
                 f"Name: {myblob.name}\n"
                 f"Blob Size: {myblob.length} bytes")

    logging.info("Triggering Queue")
    trigger_queue(myblob.name)

def trigger_queue(blob_name):
    queue_service = QueueService(connection_string=os.environ['AzureStorageQueuesConnectionString'])
    queue_service.encode_function = QueueMessageFormat.text_base64encode
    queue_service.put_message('queuetriggertesting', blob_name)

This function never gets triggered. I've got a similar function running off of a queue trigger:

import logging
import azure.functions as func

def main(msg: func.QueueMessage) -> None:
    logging.info('Python queue trigger function processed a queue item: %s',
                 msg.get_body().decode('utf-8'))
{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "name": "msg",
      "type": "queueTrigger",
      "direction": "in",
      "queueName": "testingqueue",
      "connection": "AzureStorageQueuesConnectionString"
    }
  ]
}

It is successfully connected to the azurite queue and polling for messages. The storage container on the same azurite instance is not registering, however.

AdamUnderwood commented 5 years ago

I switched to the old image arafato/azurite and the above code works. I'm not sure what has changed since it was ported over. One thing I've noticed that is different between 2 two images, is that when I connect via storage explorer, on the old image it connects fine. On the latest image, it complains similar to this post: https://social.msdn.microsoft.com/Forums/en-US/fd50c41b-94ce-465e-b905-806f0ee862b3/mac-unable-to-retrieve-child-resources-details-container-name-format-is-incorrect?forum=windowsazuredata Is the image coming preloaded with a container that doesn't conform to naming schemas?

I'll follow this thread, but for now will stick to using the older version. Thanks for your help. Hopefully the above might help uncover the issue (if there even is an issue).

mydiemho commented 5 years ago

I'll follow this thread, but for now will stick to using the older version. Thanks for your help. Hopefully the above might help uncover the issue (if there even is an issue).

I'm assuming it's a similar issue to what I'm having with using the v3 version. I am not the developer, also a user like you, so hoping the dev team can give us some answer.

XiaoningLiu commented 5 years ago

Thanks guys, we recently busy working with providing queue support in V3. This is in our investigation backlog, but didn't got time for detailed look.

XiaoningLiu commented 4 years ago

Update some progress here. We recentlly released 3.3.0-preview. Can you guys try latest released 3.3.0-preview? We did some refactor for lease implementation for blobs and containers. If the issue still happens for 3.3.0-preview, please share your environments, example code, and steps for the reproduce. @blueww is helping with Azure Function comptiability invesitgation with Azurite, and we'd like more information.

Valansch commented 3 years ago

This is old, but still open, so I will share my findings. I had a similar issue, where the blob trigger function would fail to start and give "Microsoft.WindowsAzure.Storage: The specified container does not exist."

It has to do with the function runner for the blob trigger expecting a $blob container. Azurite did on initialization. There are Actually 3 requests to azurite:

  1. http://127.0.0.1/devstoreaccount1?restype=service&comp=properties
  2. http://127.0.0.1/devstoreaccount1/$logs?restype=container&comp=list&prefix=blob%2F2021%2F01%2F06%2F1900&include=metadata
  3. http://127.0.0.1/devstoreaccount1/myblob?restype=container&comp=list&maxresults=10000

After creating the $logs blob the function starts up and triggers fine on myblob.

Please tell me if you consider this an azurite issue or azure function issue. I can then create a standalone issue.

Azurite logs:

2021-01-06T19:28:31.924Z     info: Azurite Blob service is starting on 127.0.0.1:10000
2021-01-06T19:28:31.925Z     info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2021-01-06T19:28:31.925Z     info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2021-01-06T19:28:31.941Z     info: BlobGCManager:start() Starting BlobGCManager. Set status to Initializing.
2021-01-06T19:28:31.941Z     info: BlobGCManager:start() Trigger mark and sweep loop. Set status to Running.
2021-01-06T19:28:31.941Z     info: BlobGCManager:markSweepLoop() Start next mark and sweep.
2021-01-06T19:28:31.941Z     info: BlobGCManager:markSweep() Get all extents.
2021-01-06T19:28:31.944Z     info: BlobGCManager:start() BlobGCManager successfully started.
2021-01-06T19:28:31.946Z     info: BlobGCManager:markSweep() Got 1 extents.
2021-01-06T19:28:31.946Z     info: BlobGCManager:markSweep() Get referred extents.
2021-01-06T19:28:31.946Z     info: BlobGCManager:markSweep() Got referred extents, unreferenced extents count is 1.
2021-01-06T19:28:31.947Z     info: BlobGCManager:markSweep() Try to delete function entries() { [native code] } unreferenced extents.
2021-01-06T19:28:31.948Z     debug: FSExtentStore:deleteExtents() Delete extent:18a9a258-e30c-4dbc-a68f-7839a820abde location:Default path:/home/user/c/myproject/azurite/__blobstorage__/18a9a258-e30c-4dbc-a68f-7839a820abde
2021-01-06T19:28:31.950Z     info: Azurite Blob service successfully listens on http://127.0.0.1:10000
2021-01-06T19:28:31.951Z     info: BlobGCManager:markSweep() Deleted unreferenced 1 extents, after excluding active write extents.
2021-01-06T19:28:31.951Z     info: BlobGCManager:markSweepLoop() Mark and sweep finished, taken 10ms.
2021-01-06T19:28:31.951Z     info: BlobGCManager:markSweepLoop() Sleep for 600000ms.
2021-01-06T19:28:51.888Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1?restype=service&comp=properties RequestHeaders:{"host":"127.0.0.1:10000","x-ms-client-request-id":"2a9aae66-30fe-4f1c-8fb3-3bccf2f0e196","user-agent":"Azure-Storage/9.3.1 (.NET Core)","x-ms-version":"2018-03-28","x-ms-date":"Wed, 06 Jan 2021 19:28:51 GMT","authorization":"SharedKey devstoreaccount1:uGSOSAbCWNvUt+ZoXWPUIVyFs1wwsF3iU46s3l073Og="} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2021-01-06T19:28:51.889Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=undefined Blob=
2021-01-06T19:28:51.889Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a verbose: DispatchMiddleware: Dispatching request...
2021-01-06T19:28:51.892Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: DispatchMiddleware: Operation=Service_GetProperties
2021-01-06T19:28:51.893Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2021-01-06T19:28:51.893Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: PublicAccessAuthenticator:validate() Start validation against public access.
2021-01-06T19:28:51.893Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a debug: PublicAccessAuthenticator:validate() Getting account properties...
2021-01-06T19:28:51.893Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: undefined, blob: 
2021-01-06T19:28:51.893Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: PublicAccessAuthenticator:validate() Skip public access authentication. Container name is undefined.
2021-01-06T19:28:51.893Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2021-01-06T19:28:51.897Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:2a9aae66-30fe-4f1c-8fb3-3bccf2f0e196\nx-ms-date:Wed, 06 Jan 2021 19:28:51 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1\ncomp:properties\nrestype:service"
2021-01-06T19:28:51.898Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:uGSOSAbCWNvUt+ZoXWPUIVyFs1wwsF3iU46s3l073Og=
2021-01-06T19:28:51.898Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2021-01-06T19:28:51.898Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a verbose: DeserializerMiddleware: Start deserializing...
2021-01-06T19:28:51.900Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: HandlerMiddleware: DeserializedParameters={"options":{"requestId":"2a9aae66-30fe-4f1c-8fb3-3bccf2f0e196"},"restype":"service","comp":"properties","version":"2018-03-28"}
2021-01-06T19:28:51.901Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a verbose: SerializerMiddleware: Start serializing...
2021-01-06T19:28:51.908Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a debug: Serializer: Raw response body string is <?xml version="1.0" encoding="UTF-8" standalone="yes"?><StorageServiceProperties><Logging><Version>1.0</Version><Delete>true</Delete><Read>true</Read><Write>true</Write><RetentionPolicy><Enabled>false</Enabled></RetentionPolicy></Logging><HourMetrics><Version>1.0</Version><Enabled>false</Enabled><RetentionPolicy><Enabled>false</Enabled></RetentionPolicy></HourMetrics><MinuteMetrics><Version>1.0</Version><Enabled>false</Enabled><RetentionPolicy><Enabled>false</Enabled></RetentionPolicy></MinuteMetrics><Cors/><DefaultServiceVersion>2020-04-08</DefaultServiceVersion><StaticWebsite><Enabled>false</Enabled></StaticWebsite></StorageServiceProperties>
2021-01-06T19:28:51.908Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: Serializer: Start returning stream body.
2021-01-06T19:28:51.908Z 5a2f6f1a-8a29-4750-a4e1-0bf451ce622a info: EndMiddleware: End response. TotalTimeInMS=20 StatusCode=200 StatusMessage=OK Headers={"server":"Azurite-Blob/3.10.0","x-ms-client-request-id":"2a9aae66-30fe-4f1c-8fb3-3bccf2f0e196","x-ms-request-id":"5a2f6f1a-8a29-4750-a4e1-0bf451ce622a","x-ms-version":"2020-04-08","content-type":"application/xml"}
2021-01-06T19:28:52.511Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/$logs?restype=container&comp=list&prefix=blob%2F2021%2F01%2F06%2F1900&include=metadata RequestHeaders:{"host":"127.0.0.1:10000","x-ms-client-request-id":"a4c37d51-d7df-40fd-8c93-c6f5ff67ae99","user-agent":"Azure-Storage/9.3.1 (.NET Core)","x-ms-version":"2018-03-28","x-ms-date":"Wed, 06 Jan 2021 19:28:52 GMT","authorization":"SharedKey devstoreaccount1:S6zZYcCJ1Y32En0EUp/5ixMzmze8y21+alfxudVn650="} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2021-01-06T19:28:52.511Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=$logs Blob=
2021-01-06T19:28:52.511Z e032beac-a1b6-49e5-846c-70a53ea4a966 verbose: DispatchMiddleware: Dispatching request...
2021-01-06T19:28:52.511Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: DispatchMiddleware: Operation=Container_ListBlobFlatSegment
2021-01-06T19:28:52.512Z e032beac-a1b6-49e5-846c-70a53ea4a966 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2021-01-06T19:28:52.512Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: PublicAccessAuthenticator:validate() Start validation against public access.
2021-01-06T19:28:52.512Z e032beac-a1b6-49e5-846c-70a53ea4a966 debug: PublicAccessAuthenticator:validate() Getting account properties...
2021-01-06T19:28:52.512Z e032beac-a1b6-49e5-846c-70a53ea4a966 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: $logs, blob: 
2021-01-06T19:28:52.513Z e032beac-a1b6-49e5-846c-70a53ea4a966 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container $logs
2021-01-06T19:28:52.513Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2021-01-06T19:28:52.513Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:a4c37d51-d7df-40fd-8c93-c6f5ff67ae99\nx-ms-date:Wed, 06 Jan 2021 19:28:52 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/$logs\ncomp:list\ninclude:metadata\nprefix:blob/2021/01/06/1900\nrestype:container"
2021-01-06T19:28:52.513Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:S6zZYcCJ1Y32En0EUp/5ixMzmze8y21+alfxudVn650=
2021-01-06T19:28:52.513Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2021-01-06T19:28:52.513Z e032beac-a1b6-49e5-846c-70a53ea4a966 verbose: DeserializerMiddleware: Start deserializing...
2021-01-06T19:28:52.514Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: HandlerMiddleware: DeserializedParameters={"options":{"prefix":"blob/2021/01/06/1900","include":["metadata"],"requestId":"a4c37d51-d7df-40fd-8c93-c6f5ff67ae99"},"restype":"container","comp":"list","version":"2018-03-28"}
2021-01-06T19:28:52.515Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2021-01-06T19:28:52.515Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: ErrorName=StorageError ErrorMessage=The specified container does not exist.  ErrorHTTPStatusCode=404 ErrorHTTPStatusMessage=The specified container does not exist. ErrorHTTPHeaders={"x-ms-error-code":"ContainerNotFound","x-ms-request-id":"e032beac-a1b6-49e5-846c-70a53ea4a966"} ErrorHTTPBody="<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>ContainerNotFound</Code>\n  <Message>The specified container does not exist.\nRequestId:e032beac-a1b6-49e5-846c-70a53ea4a966\nTime:2021-01-06T19:28:52.514Z</Message>\n</Error>" ErrorStack="StorageError: The specified container does not exist.\n    at Function.getContainerNotFound (/usr/local/lib/node_modules/azurite/dist/src/blob/errors/StorageErrorFactory.js:14:16)\n    at LokiBlobMetadataStore.checkContainerExist (/usr/local/lib/node_modules/azurite/dist/src/blob/persistence/LokiBlobMetadataStore.js:559:49)\n    at ContainerHandler.listBlobFlatSegment (/usr/local/lib/node_modules/azurite/dist/src/blob/handlers/ContainerHandler.js:357:34)\n    at /usr/local/lib/node_modules/azurite/dist/src/blob/generated/middleware/HandlerMiddlewareFactory.js:58:18\n    at /usr/local/lib/node_modules/azurite/dist/src/blob/generated/ExpressMiddlewareFactory.js:77:63\n    at Layer.handle [as handle_request] (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:317:13)\n    at /usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:284:7\n    at Function.process_params (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:335:12)\n    at next (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:275:10)"
2021-01-06T19:28:52.515Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Set HTTP code: 404
2021-01-06T19:28:52.515Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Set HTTP status message: The specified container does not exist.
2021-01-06T19:28:52.515Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Set HTTP Header: x-ms-error-code=ContainerNotFound
2021-01-06T19:28:52.516Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Set HTTP Header: x-ms-request-id=e032beac-a1b6-49e5-846c-70a53ea4a966
2021-01-06T19:28:52.516Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Set content type: application/xml
2021-01-06T19:28:52.516Z e032beac-a1b6-49e5-846c-70a53ea4a966 error: ErrorMiddleware: Set HTTP body: "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>ContainerNotFound</Code>\n  <Message>The specified container does not exist.\nRequestId:e032beac-a1b6-49e5-846c-70a53ea4a966\nTime:2021-01-06T19:28:52.514Z</Message>\n</Error>"
2021-01-06T19:28:52.516Z e032beac-a1b6-49e5-846c-70a53ea4a966 info: EndMiddleware: End response. TotalTimeInMS=5 StatusCode=404 StatusMessage=The specified container does not exist. Headers={"server":"Azurite-Blob/3.10.0","x-ms-error-code":"ContainerNotFound","x-ms-request-id":"e032beac-a1b6-49e5-846c-70a53ea4a966","content-type":"application/xml"}
2021-01-06T19:28:52.520Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/myblob?restype=container&comp=list&maxresults=10000 RequestHeaders:{"host":"127.0.0.1:10000","x-ms-client-request-id":"8d30cf00-4bff-4131-965f-3b39bc49a16b","user-agent":"Azure-Storage/9.3.1 (.NET Core)","x-ms-version":"2018-03-28","x-ms-date":"Wed, 06 Jan 2021 19:28:52 GMT","authorization":"SharedKey devstoreaccount1:zyOLr/bYPGWaSc1YIivKfsh7C7JsX9L9tWTYPQhn5so="} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=myblob Blob=
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 verbose: DispatchMiddleware: Dispatching request...
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: DispatchMiddleware: Operation=Container_ListBlobFlatSegment
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: PublicAccessAuthenticator:validate() Start validation against public access.
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 debug: PublicAccessAuthenticator:validate() Getting account properties...
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: myblob, blob: 
2021-01-06T19:28:52.521Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container myblob
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:8d30cf00-4bff-4131-965f-3b39bc49a16b\nx-ms-date:Wed, 06 Jan 2021 19:28:52 GMT\nx-ms-version:2018-03-28\n/devstoreaccount1/devstoreaccount1/myblob\ncomp:list\nmaxresults:10000\nrestype:container"
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:zyOLr/bYPGWaSc1YIivKfsh7C7JsX9L9tWTYPQhn5so=
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 verbose: DeserializerMiddleware: Start deserializing...
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: HandlerMiddleware: DeserializedParameters={"options":{"maxresults":10000,"include":[],"requestId":"8d30cf00-4bff-4131-965f-3b39bc49a16b"},"restype":"container","comp":"list","version":"2018-03-28"}
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: ErrorName=StorageError ErrorMessage=The specified container does not exist.  ErrorHTTPStatusCode=404 ErrorHTTPStatusMessage=The specified container does not exist. ErrorHTTPHeaders={"x-ms-error-code":"ContainerNotFound","x-ms-request-id":"2043ab91-0a53-4b05-9e6b-83a6c2c3ee68"} ErrorHTTPBody="<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>ContainerNotFound</Code>\n  <Message>The specified container does not exist.\nRequestId:2043ab91-0a53-4b05-9e6b-83a6c2c3ee68\nTime:2021-01-06T19:28:52.522Z</Message>\n</Error>" ErrorStack="StorageError: The specified container does not exist.\n    at Function.getContainerNotFound (/usr/local/lib/node_modules/azurite/dist/src/blob/errors/StorageErrorFactory.js:14:16)\n    at LokiBlobMetadataStore.checkContainerExist (/usr/local/lib/node_modules/azurite/dist/src/blob/persistence/LokiBlobMetadataStore.js:559:49)\n    at ContainerHandler.listBlobFlatSegment (/usr/local/lib/node_modules/azurite/dist/src/blob/handlers/ContainerHandler.js:357:34)\n    at /usr/local/lib/node_modules/azurite/dist/src/blob/generated/middleware/HandlerMiddlewareFactory.js:58:18\n    at /usr/local/lib/node_modules/azurite/dist/src/blob/generated/ExpressMiddlewareFactory.js:77:63\n    at Layer.handle [as handle_request] (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:317:13)\n    at /usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:284:7\n    at Function.process_params (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:335:12)\n    at next (/usr/local/lib/node_modules/azurite/node_modules/express/lib/router/index.js:275:10)"
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Set HTTP code: 404
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Set HTTP status message: The specified container does not exist.
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Set HTTP Header: x-ms-error-code=ContainerNotFound
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Set HTTP Header: x-ms-request-id=2043ab91-0a53-4b05-9e6b-83a6c2c3ee68
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Set content type: application/xml
2021-01-06T19:28:52.522Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 error: ErrorMiddleware: Set HTTP body: "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n  <Code>ContainerNotFound</Code>\n  <Message>The specified container does not exist.\nRequestId:2043ab91-0a53-4b05-9e6b-83a6c2c3ee68\nTime:2021-01-06T19:28:52.522Z</Message>\n</Error>"
2021-01-06T19:28:52.523Z 2043ab91-0a53-4b05-9e6b-83a6c2c3ee68 info: EndMiddleware: End response. TotalTimeInMS=3 StatusCode=404 StatusMessage=The specified container does not exist. Headers={"server":"Azurite-Blob/3.10.0","x-ms-error-code":"ContainerNotFound","x-ms-request-id":"2043ab91-0a53-4b05-9e6b-83a6c2c3ee68","content-type":"application/xml"}
2021-01-06T19:28:55.161Z     info: Azurite Blob service is closing...
2021-01-06T19:28:55.162Z     info: BlobGCManager:close() Start closing BlobGCManager. Set status to Closing.
2021-01-06T19:28:55.162Z     info: BlobGCManager:start() Mark and sweep loop is closed.
2021-01-06T19:28:55.162Z     info: BlobGCManager:close() BlobGCManager successfully closed. Set status to Closed.
2021-01-06T19:28:55.163Z     info: Azurite Blob service successfully closed
XiaoningLiu commented 3 years ago

@Val Thanks for the finding that blob trigger function dependency on $log container. This can help others who also have the same issue.

$log container is auto created only after enabling account's log metrics.

What Azurite can do is to auto create $log container once after SetBlobServiceProperties enables service logging. But by default, $log will not be created. I'm not sure whether blob trigger function has assumption that $log always exist or not. If it has the assumption, then it needs a change too.

Feel free to open issue for Azurite or make contribution to Azurite!

SeanFeldman commented 3 years ago

@Valansch did you end up raising a separate issue?

adammartin-mednax commented 3 years ago

@XiaoningLiu I'm trying to see if enabling the blog service logging will allow for this to work but when I try to enable logging on azurite: az storage logging update --log w --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;" --services b --retention 1

I get the following error:

Client-Request-ID=d83a454a-3b11-11ec-86bb-acde48001122 Retry policy did not allow for a retry: Server-Timestamp=Mon, 01 Nov 2021 12:47:17 GMT, HTTP status code=400, Exception=Bad Request. Bad Request

So it seems like a few things:

  1. Blob Storage Triggers likely do depend on $Log
  2. Azurite does not have this turned on by default

My suggestion ... allow us to pass some start up commands to Azurite that allows us to start Azurite with the logging service enabled. The default behavior in the cloud is that logging is enabled by default so that makes sense for it to be a behavior of an emulator.

The value of this is huge. If we can get Blob Storage Triggers to work it is possible to have an entirely local function development and testing environment. This makes standing up Blob Stores and Functions massively more reliable. Can't do that without some help from the Azurite peeps ;)

naavis commented 2 years ago

Are there no new developments on this? I am having a similar issue where BlobTriggers simply do not fire when using Azurite. Even creating a blob container called $logs as described above did not help.

blueww commented 2 years ago

@naavis Thanks for bringing it up, sorry we don't have an accurate date for this feature yet : (

naavis commented 2 years ago

That's alright. Do you have any suggested workaround?

naavis commented 2 years ago

Now I must apologize, I think it does work after all. I had the blob triggers set up incorrectly. Sorry for the confusion! Since it seems to work now, maybe it's fine to close this issue, unless someone still has trouble with it.

saurabh-mridul commented 2 years ago

I also was facing similar issue but after downgrading "Microsoft.Azure.WebJobs.Extensions.Storage" nuget package from 5.0.0 to 4.0.5, blob trigger started working with azurite.

Issue still persists with latest nuget package which is required for blob trigger.

mturczyn commented 2 years ago

I am also facing the issue, is there any chance it will be resolved?

MrBildo commented 1 year ago

I'm running into this issue. I'm trying to test an Azure Timer Function. I have a container running azurite. I can connect no problem via storage explorer using the same connection string, but get the Service request failed. Status: 404 (The specified container does not exist.) exception.

Pretty much what everyone else here ran into. Did anyone ever come up with a work around?

I feel like testing timer functions against azurite in a container is a pretty common use case. :\

blueww commented 1 year ago

@MrBildo

Would you please share the Azurite debug log for "I can connect no problem via storage explorer", and "I can connect no problem via storage explorer"? Then we can look into it.

michelle-wuyn commented 1 month ago

Now I must apologize, I think it does work after all. I had the blob triggers set up incorrectly. Sorry for the confusion! Since it seems to work now, maybe it's fine to close this issue, unless someone still has trouble with it.

@naavis Could you share how you were able to manually create the $logs blob container? I tried creating in azure storage explorer but it's not allowing me to because of the special character. I also tried enabling logging on azurite, but it didn't create the logs container either.

EmmaZhu commented 1 month ago

Hi @michelle-wuyn ,

Azure Storage will create a $log container when client side Diagnostic is enabled on the account like:

image

naavis commented 1 month ago

@EmmaZhu I think @michelle-wuyn was talking about Azurite, not real Azure Storage.

I do not remember anymore how I created the container, this was over two years ago. Likely Azure Storage Explorer allowed me to create it back then or something.