[BUG] Function occasionally hangs with no error reported and timeouts

[ajstewart]

Hello, I have a Queue triggered Azure Function that performs some light processing on a single image file that should take no more than about 10-30 seconds.

I am using the v1 python programming model (due to some bindings not being ready yet that I'd like to use), and it's all synchronous.

It's using a consumption plan on linux.

Nearly every invocation goes through fine without any issue. But on multiple occurrences a day, suddenly the function just hangs and timeouts. There is no other error, it just stops, with the next message being the function timeout (it sounds very much like what happens in #1213).

Then on the queue message retry that happens because of the failure it will go through without a problem.

I'm unable to copy code here, but for context the functions process looks like:

• Storage Queue trigger
• Saves blob using blob binding and queue message url to a temporary path.
• Performs an external api request to external gather more information on the image.
• Runs process 1 on image - sometimes hangs here.
• Runs process 2 on image - most often hangs here.
• Saves new blob to a different blob storage (using sdk).
• Uploads a row to table storage.
• Sends out an event grid message using the event grid binding.

It is not a high rate of triggers, only around 10-20 an hour say.

It always consistently hangs on one of those steps above. I thought it was something I'm doing but considering it works most of the time, maybe I am not accounting for something that's happening that I'm not aware of or how the code is being executed. Really I have no idea and it's driving me a bit nuts 😅

Investigative information

Failure Example Details:

• Timestamp: 5/25/2023, 5:16:13.5130097 AM UTC
• Invocation id: 839f7c05-f7d5-455f-b81f-a359ed971b37
• Core Tools version: 4.0.5174
• Python: 3.9

Repro steps

I am unable to reproduce this locally, all my tests go through ok

Expected behavior

Function runs without hanging and timing out (as it does most of the time).

Actual behavior

Hangs occasionally, approx 3 - 10 times a day.

Related information

I realise I haven't provided much information here, I'm happy to provide more but will have to do so outside of this public issue.

[bhagyshricompany]

Thanks for reporting this issue. you can share the repro steps here: https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately pls provide the function name.region etc for checking.

[ajstewart]

Turns out I can self-close this issue, after much hunting around and experimenting and reading this page again on how synchronous functions are run: https://learn.microsoft.com/en-us/azure/azure-functions/python-scale-performance-reference

it suspiciously looked like thread locking, so I added

"PYTHON_THREADPOOL_THREAD_COUNT" = 1

to the app settings and sure enough that has fixed the problem. I experimented by removing it and the timeouts returned. I've now had multiple days of no timeouts with forcing only 1 thread.

Why the thread locking occurs I'm not sure, maybe something to do with a scikit-learn method I am using as that seems to be the spot it consistently broke.

@bhagyshricompany would you still like to know the details to investigate? I did include an invocation id above, can you use that? The region is uk west.