[BUG] FunctionApp failed to boot citing "RuntimeError: Cannot enter into task"

[nokusukun]

Investigative information

On 2023-03-04 at 09:15:58Z, our Azure FunctionApp failed to start up with the following error:

The error appeared out of nowhere, without any changes made to the FunctionApp configuration or code. Our FunctionApp is integrated with a CI workflow from GitHub.

It is worth mentioning that an identical staging environment works fine, and the test environment also works correctly. We attempted to revert the build to a previous version from the past few days, but the error persists.

2023-03-04T09:15:58Z   [Verbose]   Request successfully matched the route with name 'function' and template '{*route}'
2023-03-04T09:15:58Z   [Information]   Executing 'Functions.function' (Reason='This function was programmatically called via the host APIs.', Id=a635f251-5cb0-4dad-a489-482a86cdd7b3)
2023-03-04T09:15:58Z   [Verbose]   Sending invocation id:a635f251-5cb0-4dad-a489-482a86cdd7b3
2023-03-04T09:15:58Z   [Verbose]   Posting invocation id:a635f251-5cb0-4dad-a489-482a86cdd7b3 on workerId:50fea28c-2c52-4041-bdca-6a3839b298e3
2023-03-04T09:15:58Z   [Error]   Exception in callback <TaskStepMethWrapper object at 0x70faf4596e30>()
handle: <Handle <TaskStepMethWrapper object at 0x70faf4596e30>()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
RuntimeError: Cannot enter into task <ContextEnabledTask pending name='Task-6' coro=<Dispatcher._dispatch_grpc_request() running at /azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py:238>> while another task <ContextEnabledTask pending name='Task-4' coro=<Dispatcher._dispatch_grpc_request() running at /azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py:249>> is being executed.

• Timestamp: (First Reported Outage) 2023-03-04T03:04:47.920Z
• Function App name: pecado-wallet-production-v2
• Function name(s) (as appropriate): function
• Core Tools version: 4

Repro steps

Steps to Reproduce:

Deploy Azure FunctionApp from GitHub. Attempt to start the FunctionApp. Observe the error message. Expected Result:

The Azure FunctionApp should start up successfully without any errors.

Actual Result:

The Azure FunctionApp fails to start up, and the error message "Cannot enter into task" is displayed.

Impact:

The Azure FunctionApp is currently inaccessible, which is affecting our services' availability.

Workaround:

We attempted to revert to a previous build, but the error persists. There are no known workarounds at the moment. The only workaround so far is to rebuild an identical environment and switch everything to that.

Next Steps:

We are currently investigating the issue further and will update this bug report as we gather more information.

Contents of the requirements.txt file:

# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues

azure-functions==1.11.0
web3==5.28.0
fastapi~=0.85.1
nest_asyncio==1.5.6

requests~=2.28.1
starlette~=0.20.4
ratelimit~=2.2.1
pyjwt~=2.6.0
bcrypt~=4.0.1
pymongo~=4.3.3
logbook~=1.5.3
retry~=0.9.2

# KYC
azure-storage-blob
azure-identity

# Email
azure-communication-email

# Wallet
pycryptodome
pysha3
pydantic~=1.10.2
hexbytes~=0.3.0
pandas~=1.5.2
fredapi~=0.5.0
cachetools~=5.3.0

# Swize
alpha_vantage

# OTP
pyotp~=2.8.0

[bhagyshricompany]

Thanks for informing will check and update

[bhagyshricompany]

You can check this below step to solve the problems. Restart the FunctionApp to see if the issue persists. Check the Azure FunctionApp logs to see if there are any other error messages or warnings that could be related to this issue. Review any recent changes to the code or environment to see if there were any updates that could have caused this issue. Verify that all dependencies and modules are up to date and compatible with the current version of Python. Try deploying the code to a different environment to see if the issue persists. If the issue does not occur in the new environment, it may indicate that there is an issue with the current environment.

[nokusukun]

Hi, we've tried restarting the FunctionApp. As for the errors, that's the only error that shows up in the logs. The issue randomly popped up after cold booting, the last CI movement was 12 or so hours before the problem started, before that the application was functioning properly.

From the last python dependency changes, there was only one package that has been added which is pyotp.

We've tried reverting the build successively until it was 4 versions behind mainline.

Redeploying the application on a fresh FunctionApp instance produced no issues.

The docker logs didn't indicate any other issues other than the app failing to respond due to the above issue.

[miriess]

Also happened to one of our environments: Two functions no longer responding and deleting and rebuilding them does not fix the issue. Is there any update on fixing this BESIDES killing the whole environment? How can Azure functions randomly be destroyed with NO hint on the issue within more than a week?

[bhagyshricompany]

Hi @miriess Thanks for input pls share the function app name ,Time Instance ,InstanceId etc.

[miriess]

After contacting support, the issue for us seems to have been connected to the "handle" method of AsgiHandler which is deprecated even though there is little evidence of it outside of the source code.

Replacing func.AsgiMiddleware(app).handle(req,context) for await func.AsgiMiddleware(app).handle_async(req,context) fixed it for us. Theory is, that 4.11 is slowly deployed in the workers and it does not work well with "handle" which is why only some of our functions failed. But that is unproven.

[gavin-aguiar]

@nokusukun have you tried using handle_async(). Handle_async was added since the async calls were not working as expected detailed in this issue. From your requirements.txt, you have pinned the azure.function library to 1.11.0. Could you use the latest azure.functions library so that you have all the latest updates and fixes. The handle_async method is supported for version >=1.12.0

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[nokusukun]

After contacting support, the issue for us seems to have been connected to the "handle" method of AsgiHandler which is deprecated even though there is little evidence of it outside of the source code.

Replacing func.AsgiMiddleware(app).handle(req,context) for await func.AsgiMiddleware(app).handle_async(req,context) fixed it for us. Theory is, that 4.11 is slowly deployed in the workers and it does not work well with "handle" which is why only some of our functions failed. But that is unproven.

Sorry for necroposting but this came up again after we decided to rebuild the functions.

This suggestion worked for us after revisiting. Thank you!