search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.92k stars 442 forks source link

Python language worker crashes when executing functions #3549

Open balag0 opened 5 years ago

balag0 commented 5 years ago

All requests to the container failed with the below exception.

West US FunctionsLogs | where RoleInstance contains "171AE528-636740280065455456" | where TIMESTAMP >= datetime(2018-10-01 23:23:00.0000000) | where TIMESTAMP <= datetime(2018-10-01 23:23:00.0000000) + 30m | where Tenant == "fe133483000c4137a26aabfc6365308c" | where SourceNamespace == "LWAWS" | where Level < 4

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.scalethis ---\u003e Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerChannelException : Failed to start language worker for: python ---\u003e System.AggregateException : One or more errors occurred. (bash exited with code 137 ) (bash exited with code 137 ) (bash exited with code 137 ) ---\u003e bash exited with code 137 ---\u003e (Inner Exception #0) Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 \u003c--- ---\u003e (Inner Exception #1) Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 \u003c--- ---\u003e (Inner Exception #2) Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException : bash exited with code 137 \u003c--- End of inner exception at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/Rpc/WorkerLanguageInvoker.cs : 74 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionInvokerBase.cs : 84 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task1 src) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionGenerator.cs : 225 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 63 at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs : 63 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task1 src) at /azure-functions-host-26390c79fe40b04617bc7fc3a9add054dec27310/src/WebJobs.Script/Description/FunctionGenerator.cs : 225 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 508 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 444 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 249 End of inner exception at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 293 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 89

asavaritayal commented 5 years ago

@balag0 can you share steps for repro?

balag0 commented 5 years ago

https://github.com/Azure/azure-functions-host/issues/3551#issuecomment-436737862 applies here also.

joal42 commented 5 years ago

how did you get it passed? I have the same issue. Function will crash after ~1hour with same error. My function runs at docker locally triggered by eventhub and output to cosmosdb.