Azure / azure-functions-python-worker

Python worker for Azure Functions.
http://aka.ms/azurefunctions
MIT License
331 stars 100 forks source link

[BUG] Init Indexing leads to exception being not outputted on timeout #1496

Open tspring418 opened 1 month ago

tspring418 commented 1 month ago

Investigative information

Please provide the following:

Repro steps

Provide the steps required to reproduce the problem:

No repro at the moment, but basically since the push out of : PYTHON_ENABLE_INIT_INDEXING

We have seen an issue that has led to the exceptions during init leading to ten second timeout between host and language worker prevents logging of the exception causing the problem during this step. Which leads to confusion on why a failure may be occuring during startup of the language worker.

Indexing times using old indexing

 

percentile_AverageTime_30 | percentile_AverageTime_50 | percentile_AverageTime_75 | percentile_AverageTime_90 | percentile_AverageTime_95 | percentile_AverageTime_99 -- | -- | -- | -- | -- | -- 16.5 | 16.5 | 16.6 | 21.406 | 21.406 | 21.406

 

Expected behavior

Metadata call during Indexing should occur while if a timeout happens, it will return the exception that occurred during this time that leads to the timeout and log that exception as well.

Actual behavior

Only a timeout exception is shown System.TimeoutException : The operation has timed out. at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout() at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1764

While the captured exception isn't logged out during indexing.

Known workarounds

Provide a description of any known workarounds.

PYTHON_ENABLE_INIT_INDEXING = 0 in app settings will allow you to see the real exception of the failure

Contents of the requirements.txt file:

Provide the requirements.txt file to help us find out module related issues.

n/a

Related information

Provide any related information

(https://github.com/Azure/azure-functions-python-worker/blob/5bc5d39efffe86da25cacdf7d612a2bc03b32ec3/azure_functions_worker/constants.py#L68C32-L68C59)

ICM # 503543832

Source

-->