Azure / azure-functions-python-worker

Python worker for Azure Functions.
http://aka.ms/azurefunctions
MIT License
335 stars 103 forks source link

[BUG] Azure Functions Python Worker does not report errors during function startup #1165

Open sapountzis opened 1 year ago

sapountzis commented 1 year ago

We have encountered an issue when using version 4 of the azure-functions-python-worker package.

The problem occured when the python function crashed during startup, due to an import error, which was breaking our code.

In this case, there is no error message in the azure logs. This makes it very difficult to identify the reason for the crash and debug it.

One sign of this issue is when the "Syncing triggers..." action fails during deployment after multiple retries.

Another sign is receiving a "no functions found" warning that has nothing to do with the error.

To reproduce the bug, you can create any function that has an import error, syntax error or any other error that prevents the function from starting.

The fundamental issue seems to be that there is no direct access to the python runtime where the app lives. The only deployment option for the "consumption" plan is ZipDeploy that builds everything in a separate environment and then transfers the files to the actual azure function environment. We accidentally managed to find the actual bug in the code that was crashing the function when we decided to use a custom docker image for our azure function app after all our previous debugging attempts failed. Inside the docker container, we executed the "func start" during build time just to test if it's working and that's where we saw the import error.

We believe that the azure logging should report the function crash in the same way that running it in the console locally does. This would greatly improve the debugging experience for developers.

We hope that this issue can be addressed in a future release of the azure-functions-python-worker package.

bhagyshricompany commented 1 year ago

Hi @sapountzis It's great that you have provided all the details about the bug and also suggested a workaround for it. This will help the Azure team to reproduce the bug and fix it in the future. will update you soon on this.

sapountzis commented 1 year ago

Reproduction example

import azure.functions as func
import logging
import not_existing_package

app = func.FunctionApp()

@app.function_name(name='test_func')
@app.schedule(schedule='0 10 4 * * *', arg_name='timer', run_on_startup=False, use_monitor=False)
async def test_func(timer: func.TimerRequest) -> None:
    logging.warning('hello')

Try this in the cloud and see if you get anything in the logs

espenairmine commented 1 year ago

This is relevant also if some other errors (code works locally) stops the function app from starting.

jpboily-axceta commented 1 year ago

We would also love this bug to be fixed. We circumvented the problem by adding a script that calls the function_app.py module directly with the python binary to print the stack trace when it can not start.

jdmargulici commented 1 month ago

@jpboily-axceta could you elaborate a bit on how to do this? I am facing this very issue with about 20 import statements and I don't know how to diagnose which one (s) is(are) the culprit.