Azure / azure-event-hubs-node

Node client library for Azure Event Hubs https://azure.microsoft.com/services/event-hubs
MIT License
50 stars 46 forks source link

Cannot read property 'address' of undefined / Detach already received / transfer after detach #205

Closed dan-v-elevate closed 5 years ago

dan-v-elevate commented 5 years ago

Describe the bug We are using this library 'azure/event-hub' in a v2 Azure function with an event hub trigger. The function essentially reads in batches of data, transforms it, and outputs it to a number of other event hubs. Note that we are using the consumption plan, so we have somewhat limited insight into the underlying hosts where this is running. The issue we are seeing results in the azure function host crashing and not recovering (opened an issue for that first https://github.com/Azure/azure-functions-host/issues/4079) and we get a very limited set of errors bubbled up that appears to possibly be related to this library. The errors: Error: Detach already received, Error: transfer after detach, and TypeError: Cannot read property 'address' of undefined is what we are seeing in the stack trace which looks pretty similar to another issue already posted here #171 . To be clear, I don't know for certain this is the issue based on the limited stack trace, but if you could provide any input for helping further debug if it is or isn't that would be really appreciated.

Stack trace from azure functions ``` [ { "severityLevel":"Error", "outerId":"0", "message":"Exception while executing function: Functions.iot_intake", "type":"Microsoft.Azure.WebJobs.Host.FunctionInvocationException", "parsedStack":[ { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":0, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__17.MoveNext", "level":1, "line":293, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":2, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":3, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":4, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":5, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__14.MoveNext", "level":6, "line":89, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" } ], "id":"5240712" }, { "severityLevel":"Error", "outerId":"5240712", "message":"Failed to start language worker for: node", "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerChannelException", "parsedStack":[ { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":0, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":1, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":2, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":3, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+d__6.MoveNext", "level":4, "line":74, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":5, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":6, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":7, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":8, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+d__24.MoveNext", "level":9, "line":84, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":10, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":11, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":12, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":13, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+d__2.MoveNext", "level":14, "line":20, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":15, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":16, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":17, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":18, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+d__9.MoveNext", "level":19, "line":63, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":20, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":21, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":22, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":23, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__25.MoveNext", "level":24, "line":556, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":25, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":26, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":27, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":28, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__24.MoveNext", "level":29, "line":503, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":30, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":31, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":32, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":33, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__23.MoveNext", "level":34, "line":439, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":35, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":36, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":37, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":38, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__17.MoveNext", "level":39, "line":249, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":40, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":41, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":42, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":43, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+d__6.MoveNext", "level":44, "line":74, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":45, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":46, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":47, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":48, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+d__24.MoveNext", "level":49, "line":84, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":50, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":51, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":52, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":53, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+d__2.MoveNext", "level":54, "line":20, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":55, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":56, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":57, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":58, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+d__9.MoveNext", "level":59, "line":63, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":60, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":61, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":62, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":63, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__25.MoveNext", "level":64, "line":556, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":65, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":66, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":67, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":68, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__24.MoveNext", "level":69, "line":503, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":70, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":71, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":72, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":73, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__23.MoveNext", "level":74, "line":439, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":75, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":76, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":77, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":78, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__17.MoveNext", "level":79, "line":249, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":80, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":81, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":82, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":83, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+d__6.MoveNext", "level":84, "line":74, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":85, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":86, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":87, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":88, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+d__24.MoveNext", "level":89, "line":84, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":90, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59468, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__24.MoveNext", "level":59469, "line":503, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59470, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59471, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59472, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59473, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__23.MoveNext", "level":59474, "line":439, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59475, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59476, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59477, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59478, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__17.MoveNext", "level":59479, "line":249, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59480, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59481, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59482, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59483, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+d__6.MoveNext", "level":59484, "line":74, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59485, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59486, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59487, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59488, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+d__24.MoveNext", "level":59489, "line":84, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59490, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59491, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59492, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59493, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+d__2.MoveNext", "level":59494, "line":20, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59495, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59496, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59497, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59498, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+d__9.MoveNext", "level":59499, "line":63, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59500, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59501, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59502, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59503, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__25.MoveNext", "level":59504, "line":556, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59505, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59506, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59507, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59508, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__24.MoveNext", "level":59509, "line":503, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59510, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59511, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59512, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59513, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__23.MoveNext", "level":59514, "line":439, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59515, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59516, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59517, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59518, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__17.MoveNext", "level":59519, "line":249, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59520, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59521, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59522, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59523, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+d__6.MoveNext", "level":59524, "line":74, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59525, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59526, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59527, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59528, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null", "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+d__24.MoveNext", "level":59529, "line":84, "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59530, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59531, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59532, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59533, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+d__2.MoveNext", "level":59534, "line":20, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59535, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59536, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59537, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59538, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+d__9.MoveNext", "level":59539, "line":63, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59540, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59541, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59542, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult", "level":59543, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__25.MoveNext", "level":59544, "line":556, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59545, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59546, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59547, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59548, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__24.MoveNext", "level":59549, "line":503, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59550, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59551, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59552, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59553, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__23.MoveNext", "level":59554, "line":439, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw", "level":59555, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess", "level":59556, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification", "level":59557, "line":0 }, { "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult", "level":59558, "line":0 }, { "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35", "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+d__17.MoveNext", "level":59559, "line":249, "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs" } ], "id":"57175771" }, { "severityLevel":"Error", "outerId":"57175771", "message":"One or more errors occurred. (node exited with code 1\n [error] Worker 2407da9e-1a4c-460f-a5c4-c34846a4f4de uncaught exception: TypeError: Cannot read property 'address' of undefined) (node exited with code 1\n [error] Worker 8d64531a-c854-41e6-b06a-a7179d8583f7 uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: transfer after detach) (node exited with code 1\n [error] Worker 2537693d-d57a-4d81-898e-952329111b3a uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: Detach already received)", "type":"System.AggregateException", "id":"45787762" }, { "severityLevel":"Error", "outerId":"45787762", "message":"node exited with code 1\n [error] Worker 2407da9e-1a4c-460f-a5c4-c34846a4f4de uncaught exception: TypeError: Cannot read property 'address' of undefined", "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException", "id":"55567947" }, { "severityLevel":"Error", "outerId":"45787762", "message":"node exited with code 1\n [error] Worker 8d64531a-c854-41e6-b06a-a7179d8583f7 uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: transfer after detach", "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException", "id":"58998898" }, { "severityLevel":"Error", "outerId":"45787762", "message":"node exited with code 1\n [error] Worker 2537693d-d57a-4d81-898e-952329111b3a uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: Detach already received", "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException", "id":"57767042" } ] ```


To Reproduce We don't have a simple reproduction of the issue at the moment, but it has happened very regularly for us over the past month. In the Azure function we establish a few connections to different event hubs (eventHubClient.createFromConnectionString) outside of the function handler and then during function executions we are sending data to these event hubs (await client.send({ body: output, partitionKey: partitionKey)). Sometimes after recycling hosts we will hit this issue immediately and other times we can go a few days without hitting it.

Expected behavior Expect connections to event hub to keep being able to send data without throwing these exceptions.

Package-name: azure/event-hubs Package-version: 1.0.8 node.js version: 8.11.1 OS name and version: Running in Azure functions consumption plan on Windows (32 bit)

Additional Information Additionally I'm also seeing the issue #186 fairly frequently and not sure if this is related/contributing to this issue.

Stack trace for InvalidOperationError ``` Timestamp: 2019-02-13T12:22:48.2390823Z Inocation ID: dc2f9399-9f5e-4a61-bc1a-23d5f7d21ccd { InvalidOperationError: Error sending batch to rules event hub -- A link to connection '237673' $cbs node has already been opened. at Object.translate (D:\home\site\wwwroot\node_modules\@azure\amqp-common\dist\lib\errors.js:527:17) at CbsClient. (D:\home\site\wwwroot\node_modules\@azure\amqp-common\dist\lib\cbs.js:87:32) at Generator.throw () at rejected (D:\home\site\wwwroot\node_modules\tslib\tslib.js:105:69) at at process._tickCallback (internal/process/next_tick.js:188:7) name: 'InvalidOperationError', translated: true, retryable: false, info: null, condition: 'amqp:not-allowed' } ```


ramya-rao-a commented 5 years ago

@dan-v-elevate Apologies for not responding sooner.

We have just released a new version 2.0.0 for the Event Hubs library which includes the fix for #186 Please see https://github.com/Azure/azure-event-hubs-node/issues/186#issuecomment-477364700

Do you still see the other errors that you mentioned in the issue description?

The TypeError: Cannot read property 'address' of undefined should have an accompanying stack trace, which will be helpful for us to figure out what the problem is.

The other errors Error: Detach already received and Error: transfer after detach are related to the underlying message transfer details in AMQP which is hard for us to debug without more logs. If you still see this after updating to version 2.0.0 of this library, then I can work with you to enable logging to understand the issue better

dan-v-elevate commented 5 years ago

@ramya-rao-a - thanks for the update on this! It seems like the majority of our issues are coming from 'A link to connection '.....' $cbs node has already been opened' issue. We will pull in 2.0 and see if that helps.

dan-v-elevate commented 5 years ago

@ramya-rao-a - just wanted to let you know that 2.0 seems to have fixed our issues. Thanks!

ramya-rao-a commented 5 years ago

That's great to hear, thanks @dan-v-elevate!