Function App returns 202 when incorrect function name configured as Event Grid webhook - should return error code instead

rodolfograve commented 2 years ago

Setup

Blob Storage trigger using Source=EventGrid (as per https://learn.microsoft.com/en-us/azure/azure-functions/functions-event-grid-blob-trigger)

This setup requires an Event Grid Subscription using an Endpoint Type = Webhook, and the URL must be in the following format: _https://.azurewebsites.net/runtime/webhooks/blobs?functionName=&code=<BLOB_EXTENSIONKEY

Issue Summary

If the __ is wrong, i.e. there is no function with that name, the Function App returns 202 to Event Grid, which causes Event Grid to consider the event delivered. However, since there is no function with that name, the Function App logs a DEBUG-level log and drops the event.

Consequences

We have irrecoverable and undetected lost events. Event Grid does not retry, and worse, does not apply any dead-lettering policies to the event.

We are also not able to see this error in the logs using the default configuration.

Via a Microsoft Support ticket (#2206230050002214) we have been told to set the logging level to DEBUG so that we can pick the "error" message from the Function App, but even though that would help us with spotting the issue as soon as possible, it still does not address the underlying issue and its consequences. Critically, we lose the ability to retry any lost events.

Investigative information

Timestamp: September 29th, 2022, 18:00 (cannot see exact timestamp via the Event Grid metrics)
Function App version: ~4
Function App name: nemsdevwebappd0anonfn
Function name(s) (as appropriate): N/A (any name that does not map to an existing function)
Invocation ID: 9fb4acad-a569-4ecb-b76c-30685e6c22a7
Region: UK South

Repro steps

Create an Event Grid Subscription using a Webhook.
Use the following format for the Webhook URL, making sure the Function App exists but not the function: https://.azurewebsites.net/runtime/webhooks/blobs?functionName=&code=<BLOB_EXTENSION_KEY
Trigger the Event Grid event so that Event Grid attempts to deliver the event to your non-existing function.

Expected behavior

The Function App should:

Return some error code (404 seems natural given that the target function was not found). This will allow Event Grid to realise the event was not delivered and trigger all its retry + dead-lettering mechanisms.
Log an ERROR-level message.

Actual behavior

Event Grid reports the event as delivered.
The event is not processed and it's lost.
There is no error log in the Function App.

There seems to be a DEBUG log entry, which is not visible using a Function App's recommended log configuration:

Known workarounds

None

Related information

Programming language used: C#
Triggers used: BlobTrigger(Source = BlobTriggerSource.EventGrid)

rodolfograve commented 2 years ago

Hello. Is this a valid channel to report this? After 8 days I haven't received even an acknowledgement.

rodolfograve commented 2 years ago

14 days and counting. Is there any other way to contact this team?

We are a paying customer and this level of support feels more appropriate for free software than a profitable company like Microsoft.

We already went through Microsoft's/Azure support channel and were sent to this GitHub repository as the only way to engage with the Product Team.

rodolfograve commented 1 year ago

I was hoping that after the .NET 7 GA someone would pick this ticket up?

rodolfograve commented 1 year ago

Echoooooooo

fabiocav commented 1 year ago

@rodolfograve apologies for the delayed response here.

For issues like this, if you're seeing production impact, and since you have access to support, using that channel would ensure the issue is handled in a timely manner.

Are you still experiencing this?

rodolfograve commented 1 year ago

Yes, unfortunately the situation hasn't changed. 7 months on and still experiencing the same behaviour.

At the time I raised a support request and spent weeks going back and forth with the support team. The outcome was "please, create a ticket in this GitHub project".

There is an obvious issue with the way problems are dealt with in Azure Functions, and I hope that by it being so obvious it can be fixed.

Or maybe I just have bad luck :-)

tlaukkanen commented 2 months ago

I think I might have this function name issue with #10406 where I have Event Grid webhook based blob handler function which starts up without issues in Functions App and my function shows up as BlobHandlerFunction in functions list. Then I have Event Grid subscription with webhook endpoint url is: .../runtime/webhooks/blobs?functionName=BlobHandlerFunction&code=...

But when blob is triggered I see this in function app log stream:

Blob 'my-folder/111_2024_08_19_10_22' will not be processed because function 'BlobHandlerFunction' cannot be found. Message Id: 'e0...75'

Question: Is there some specific format for the function name - like should it contain namespace too? On this page it has this Host.Functions. prefix in the functionName value.

My function looks like this:

namespace BlobHandler
{
    public class BlobHandler
    {
...
        [Function("BlobHandlerFunction")]
        public async Task RunAsync(
            [BlobTrigger("%ContainerName%/%BlobFolder%/{name}",
              Source = BlobTriggerSource.EventGrid,
              Connection = "AzureWebJobsStorage")] Stream stream,
            string name,
            ILogger log,
            CancellationToken cancellationToken)
        {
...

tlaukkanen commented 2 months ago

Just if someone else finds this for their problem: In our case the Event Grid webhook started to work when I changed the following:

Event Grid webhook URL to have Host.Functions. prefix for function name, like: ...?functionName=Host.Functions.BlobHandlerFunction&...

Changed the function method name to match the function name:

[Function(nameof(BlobHandlerFunction))]
public async Task BlobHandlerFunction(

Azure / azure-functions-host