[BUG] Azure Functions getting Exception while executing function: Functions.extract_and_chunk Did not find any initialized language workers

erobdxc commented 8 months ago

Investigative information

Please provide the following:

Timestamp: 01/11/2023, 10:13:52.102851
Function App name:
Function name(s) (as appropriate):
Core Tools version: 3.*, 4.0.0

Azure function using runtime version 4 and python stack and version 3.10

Repro steps

Provide the steps required to reproduce the problem:

Deployed the Python 3.10 function via a CI pipeline on GitHub Actions to Azure Functions premium and then using the test function option on Azure Portal

Expected behavior

Provide a description of the expected behavior.

The function should execute and perform the python script

EDIT: I should have add, that this function works fine on Consumption tier and only running into this issue with Premium

Actual behavior

Provide a description of the actual behavior observed.

The function returns the response Exception while executing function: Functions.extract_and_chunk Did not find any initialized language workers after around 80ms after trying to test the function.

Call Stack Logs Microsoft.Azure.WebJobs.Host.FunctionInvocationException: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108) Inner exception System.InvalidOperationException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw: at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcherLoadBalancer.GetLanguageWorkerChannel (Microsoft.Azure.WebJobs.Script, Version=4.27.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcherLoadBalancer.cs:27) at Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher+<InvokeAsync>d__59.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.27.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs:400) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.27.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:97) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.27.0.0, Culture=neutral, PublicKeyToken=null: /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:82) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2+d2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:20) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+d10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306) `

Known workarounds

Provide a description of any known workarounds.

None, Ive seen the error on multiple forms but no clear answer

Contents of the requirements.txt file:

Provide the requirements.txt file to help us find out module related issues.

Example:

azure-functions
pandas
openai
azure-storage-file-datalake
azure-storage-blob
azure-keyvault
azure-identity
pycryptodome
tiktoken
pdfplumber
python-pptx
python-docx
odfpy
striprtf
requests
pyarrow

bhagyshricompany commented 8 months ago

Thanks for reporting.Please provide the name, ID, region, timestamp, and relevant source code for the function or application you're referring to for repro.

erobdxc commented 8 months ago

Thanks for the reply, Funcation Name: altice-function-preproc-prem Region: West Europe Timestamp: 01/11/2023, 10:13:52.102851

Would the ID be the Subscription ID?

With regards to the source code due to privacy concerns I cannot share it, but it is a timer trigger and it reads secrets from a keyvault, loads data from a blob extracts data and then moves the data to another blob, but i have two functions on the one Function App that are different, but both get the same error, the other is used for embedding.

bhagyshricompany commented 8 months ago

function invocation id

erobdxc commented 8 months ago

Hi this is the ID, sorry for the delay: 2f945479-dbd9-4aa8-b172-f46c991f356d

bhagyshricompany commented 7 months ago

Hi Can you add with consumption plan with this app setting as found with log using function this is missing.https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#python_isolate_worker_dependencies.Thanks

erobdxc commented 7 months ago

Hi, thanks for the response, why for the consumption plan? That works fine, the issues i am having is with premium? I did this suggestion with my premium function and it executed as expected once and then went back to the original "Did not find any initialized language workers" error

MasterSalami commented 7 months ago

Hello, I've been running into the same issue for the past few weeks, with a very similar context:

Function (even just a simple log + return) may run successfully once, then stays in error "Did not find any initialized workers".
Premium App service plan, and cannot switch to consumption as we need Vnet integration.
Python + functions as scripts work fine when running from SSH in the "Advanced tools"

In the end, what worked was setting the App setting "WEBSITE_RUN_FROM_PACKAGE" to 1 in the Configuration. Hope that helps.

erobdxc commented 7 months ago

Thanks for that suggestion, that seems to have worked!

Azure / azure-functions-python-worker