search

Azure / azure-functions-servicebus-extension

Service Bus extension for Azure Functions
MIT License
65 stars 36 forks source link

Linux consumption based function with service bus trigger stops processing after 10 minutes #71

Closed tomgallard closed 2 months ago

tomgallard commented 4 years ago

Check for a solution in the Azure portal

My dotnetcore3.1 linux function app seems to stop processing messages after 10 minutes (which is also the timeout setting I have in host.json. I am using a zip based deployment to Azure storage referenced using the WEBSITE_RUN_FROM_PACKAGE application setting.

I am using a service bus trigger, with autocomplete set to false. I am completing/failing the messages in code, and am confident there are no code paths where messages are not being completed or failed.

After the function stops processing messages if I go back into the function on azure portal, it will restart processing messages.

image

I also never see the function scaling beyond 1 instance.

Investigative information

Please provide the following:

Repro steps

Provide the steps required to reproduce the problem:

  1. Click function name in azure portal, observe in app insights live metrics that messages are being processed. Only 1 instance appears processing messages regardless of queue length
  2. Wait 10 minutes
  3. Observe messages are no longer being processed.
  4. Click function name in azure portal. Observe messages start being processed again

Expected behavior

Provide a description of the expected behavior.

Function should continue processing messages for longer than 10 minutes.

Actual behavior

Messages stop being processed after 10 minutes

Related information

Provide any related information

Dotnetcore 3.1 Service bus binding-

tomgallard commented 4 years ago

Have now observed this happening in app insights live metrics- I see the following logs:

After this nothing more from that instance, and no other instance seems to start to replace it.

tomgallard commented 4 years ago

Further information for investigation - I believe this was caused by the user of a custom INameResolver to resolve the queue name.

Removing the use of an INameResolver seems to resolve it. But I can't find anywhere in the docs whether this is expected behaviour or not.

TimPosey2 commented 4 years ago

Having similar issues, but on Windows. No scaleout beyond 1 instance and dies after ~10 or so minutes. However I'm NOT using a custom INameResolver.

TimPosey2 commented 4 years ago

Found the issue for me thanks to Azure Support. Make sure the service bus connection string access policy has the MANAGE access turned on for it. This will allow Azure Functions to be able to view the queue and topic metrics.

image

alrod commented 4 years ago

@tomgallard,

  1. it looks like the app "v4api-pwt-multi-image-support-jobconsumer" is deleted.
  2. Custom INameResolver could cause the issue for consumption plan function app as scale controller (https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#runtime-scaling) is not aware about custom INameResolver. Only the actual function app has the custom INameResolver code.

Workaround is using AppService plan.

tomgallard commented 4 years ago

Thanks- I have worked around by not using a custom INameResolver. It is very concerning that this is not mentioned in the docs anywhere, nor are any compile time errors/warnings, or run time errors/warnings evident - it just fails silently!

alrod commented 4 years ago

@jeffhollan, for updating docs: Custom INameResolver could cause the issue for consumption plan function app as scale controller (https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#runtime-scaling) is not aware about custom INameResolver. Only the actual function app has the custom INameResolver code.