search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.94k stars 441 forks source link

deadlock in TableExtension.GetTable #1656

Open ahmelsayed opened 7 years ago

ahmelsayed commented 7 years ago

Deadlock call stack is below. I also have a memory dump. More context in an email with subject Function monitoring page stuck

    [Managed to Native Transition]  
>   mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)   Unknown
    mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Unknown
    mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)    Unknown
    mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)    Unknown
    mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Tables.TableExtension.GetTable(Microsoft.Azure.WebJobs.TableAttribute attribute)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Tables.TableExtension.BuildFromTableAttribute(Microsoft.Azure.WebJobs.TableAttribute attribute)   Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.BindingFactory.DelegateConverterBuilder<System.__Canon, System.__Canon>.Convert(System.__Canon input)    Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.PatternMatcher.Invoker<Microsoft.Azure.WebJobs.TableAttribute, Microsoft.Azure.WebJobs.IAsyncCollector<Microsoft.WindowsAzure.Storage.Table.ITableEntity>>.Work.AnonymousMethod__0(object input) Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.AsyncCollectorBindingProvider<Microsoft.Azure.WebJobs.TableAttribute, Microsoft.WindowsAzure.Storage.Table.ITableEntity>.ExactBinding<Submission#0.StorageEventTestEntity>.BuildAsync(Microsoft.Azure.WebJobs.TableAttribute attrResolved, Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context)    Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.BindingBase<Microsoft.Azure.WebJobs.TableAttribute>.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.BindingContext context)  Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.Azure.WebJobs.Host.Bindings.IValueProvider>.Start<Microsoft.Azure.WebJobs.Host.Bindings.BindingBase<Microsoft.Azure.WebJobs.TableAttribute>.<BindAsync>d__7>(ref Microsoft.Azure.WebJobs.Host.Bindings.BindingBase<Microsoft.Azure.WebJobs.TableAttribute>.<BindAsync>d__7 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.BindingBase<Microsoft.Azure.WebJobs.TableAttribute>.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.BindingContext context)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.BindCoreAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, object value, System.Collections.Generic.IDictionary<string, object> parameters)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IReadOnlyDictionary<string, Microsoft.Azure.WebJobs.Host.Bindings.IValueProvider>>.Start<Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.<BindCoreAsync>d__8>(ref Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.<BindCoreAsync>d__8 stateMachine)    Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.BindCoreAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, object value, System.Collections.Generic.IDictionary<string, object> parameters)   Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, System.Collections.Generic.IDictionary<string, object> parameters) Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IReadOnlyDictionary<string, Microsoft.Azure.WebJobs.Host.Bindings.IValueProvider>>.Start<Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.<BindAsync>d__7>(ref Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.<BindAsync>d__7 stateMachine)    Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding<System.Net.Http.HttpRequestMessage>.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, System.Collections.Generic.IDictionary<string, object> parameters) Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.BindingSource.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, Microsoft.Azure.WebJobs.Host.Protocols.FunctionStartedMessage message, Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogEntry fastItem, System.Collections.Generic.IDictionary<string, Microsoft.Azure.WebJobs.Host.Protocols.ParameterLog> parameterLogCollector, System.Diagnostics.TraceLevel functionTraceLevel, System.Threading.CancellationToken cancellationToken)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__14>(ref Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__14 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, Microsoft.Azure.WebJobs.Host.Protocols.FunctionStartedMessage message, Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogEntry fastItem, System.Collections.Generic.IDictionary<string, Microsoft.Azure.WebJobs.Host.Protocols.ParameterLog> parameterLogCollector, System.Diagnostics.TraceLevel functionTraceLevel, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance functionInstance, System.Threading.CancellationToken cancellationToken)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__12>(ref Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__12 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance functionInstance, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, System.Threading.CancellationToken cancellationToken)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.<TryExecuteAsync>d__3>(ref Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.<TryExecuteAsync>d__3 stateMachine)    Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.JobHost.CallAsyncCore(System.Reflection.MethodInfo method, System.Collections.Generic.IDictionary<string, object> arguments, System.Threading.CancellationToken cancellationToken) Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.Azure.WebJobs.JobHost.<CallAsyncCore>d__38>(ref Microsoft.Azure.WebJobs.JobHost.<CallAsyncCore>d__38 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.JobHost.CallAsyncCore(System.Reflection.MethodInfo method, System.Collections.Generic.IDictionary<string, object> arguments, System.Threading.CancellationToken cancellationToken) Unknown
    Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.JobHost.CallAsync(System.Reflection.MethodInfo method, System.Collections.Generic.IDictionary<string, object> arguments, System.Threading.CancellationToken cancellationToken) Unknown
    Microsoft.Azure.WebJobs.Script.dll!Microsoft.Azure.WebJobs.Script.ScriptHost.CallAsync(string method, System.Collections.Generic.Dictionary<string, object> arguments, System.Threading.CancellationToken cancellationToken)    Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.Azure.WebJobs.Script.ScriptHost.<CallAsync>d__70>(ref Microsoft.Azure.WebJobs.Script.ScriptHost.<CallAsync>d__70 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Script.dll!Microsoft.Azure.WebJobs.Script.ScriptHost.CallAsync(string method, System.Collections.Generic.Dictionary<string, object> arguments, System.Threading.CancellationToken cancellationToken)    Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.HandleRequestAsync(Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.<HandleRequestAsync>d__37>(ref Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.<HandleRequestAsync>d__37 stateMachine)    Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.HandleRequestAsync(Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Threading.CancellationToken cancellationToken) Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.<ProcessRequestAsync>d__5>(ref Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.<ProcessRequestAsync>d__5 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Threading.CancellationToken cancellationToken) Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync.AnonymousMethod__0(System.Net.Http.HttpRequestMessage req, System.Threading.CancellationToken ct)    Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.(ref Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d stateMachine)   Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync.AnonymousMethod__0(System.Net.Http.HttpRequestMessage req, System.Threading.CancellationToken ct)    Unknown
    Microsoft.Azure.WebJobs.Extensions.Http.dll!Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, System.Func<System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken, System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>> processRequestHandler, System.Threading.CancellationToken cancellationToken)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.<ProcessRequestAsync>d__8>(ref Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.<ProcessRequestAsync>d__8 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Extensions.Http.dll!Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, System.Func<System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken, System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>> processRequestHandler, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext controllerContext, System.Threading.CancellationToken cancellationToken)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.<ExecuteAsync>d__3>(ref Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.<ExecuteAsync>d__3 stateMachine)    Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext controllerContext, System.Threading.CancellationToken cancellationToken)   Unknown
    System.Web.Http.dll!System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1>(ref System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1 stateMachine)    Unknown
    System.Web.Http.dll!System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
    System.Net.Http.dll!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)  Unknown
    System.Web.Http.dll!System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)    Unknown
    System.Net.Http.dll!System.Net.Http.DelegatingHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.<SendAsync>d__5>(ref Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.<SendAsync>d__5 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)   Unknown
    System.Net.Http.dll!System.Net.Http.DelegatingHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.<SendAsync>d__6>(ref Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.<SendAsync>d__6 stateMachine)  Unknown
    Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
    System.Net.Http.dll!System.Net.Http.DelegatingHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)   Unknown
    System.Web.Http.dll!System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)  Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.Start<System.Web.Http.HttpServer.<SendAsync>d__0>(ref System.Web.Http.HttpServer.<SendAsync>d__0 stateMachine)  Unknown
    System.Web.Http.dll!System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)  Unknown
    System.Net.Http.dll!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)  Unknown
    System.Web.Http.WebHost.dll!System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsyncCore(System.Web.HttpContextBase contextBase)   Unknown
    mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<System.Web.Http.WebHost.HttpControllerHandler.<ProcessRequestAsyncCore>d__0>(ref System.Web.Http.WebHost.HttpControllerHandler.<ProcessRequestAsyncCore>d__0 stateMachine)    Unknown
    System.Web.Http.WebHost.dll!System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsyncCore(System.Web.HttpContextBase contextBase)   Unknown
    System.Web.Http.WebHost.dll!System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsync(System.Web.HttpContext context)   Unknown
    System.Web.dll!System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest.AnonymousMethod__0()    Unknown
    System.Web.dll!System.Web.TaskAsyncHelper.BeginTask(System.Func<System.Threading.Tasks.Task> taskFunc, System.AsyncCallback callback, object state) Unknown
    System.Web.dll!System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, object extraData)  Unknown
    System.Web.dll!System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()  Unknown
    System.Web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, ref bool completedSynchronously)  Unknown
    System.Web.dll!System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Exception error)   Unknown
    System.Web.dll!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext context, System.AsyncCallback cb)  Unknown
    System.Web.dll!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest wr, System.Web.HttpContext context)    Unknown
    System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
    System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags)   Unknown
    [Native to Managed Transition]  
    [Managed to Native Transition]  
    System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
    System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags)   Unknown
    [Native to Managed Transition]
mathewc commented 7 years ago

Thanks Ahmed. Offending line of code is here: https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Tables/TableExtension.cs#L71.

While addressing this, we should review other synchronous wait code like this.

christopheranderson commented 7 years ago

@fabiocav says this specific issue with table storage should be resolved with the dotnet core migration.

@mathewc still warns there are other places this pattern might exist

Eduardo says we should look at the number of times we see this exception.

ahmelsayed commented 7 years ago

@christopheranderson it's not an exception, just a call stack of the deadlock. we can't find how many people run into this because we need to either attach a live debugger or analyze a memory dump

MikeStall commented 7 years ago

This code is:

  var account = Task.Run(() => this._accountProvider.GetStorageAccountAsync(attribute, CancellationToken.None)).GetAwaiter().GetResult();

The extra goo there should give this the same semantics as Thread.Start/Thread.Join and ensure we're not deadlocking. We use this pattern pretty broadly, so if this doesn't work, we have a bigger problem.

MikeStall commented 7 years ago

We may be blocked waiting on the GetStorageAccountAsync() task (and not actually deadlocking).

That task eventually calls into this long-running operation here: https://github.com/Azure/azure-webjobs-sdk/blob/5ea3892dbb31182d48b8d965b9fc3df585fdc8ed/src/Microsoft.Azure.WebJobs.Host/Executors/DefaultStorageCredentialsValidator.cs#L40

mamaso commented 7 years ago

Got a new dump from this in thread Function app experienced downtime several times a day

couple user threads, ~200 waiting on GetTable, ~200 waiting on AsyncInvoker (created from AsyncConverter<Attr, CloudTable>)

I think that @MikeStall is onto something in the issue - it's possible we're stuck validating the account via http requests.

GetServicePropertiesAsync has a comment mentioning that the call can fail if the storage account name is incorrect. https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Executors/DefaultStorageCredentialsValidator.cs#L41

This may be exacerbating the problem – if we have multiple concurrent requests and validation is slow, there may be many calls to ValidateCredentialsAsync. https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Executors/DefaultStorageAccountProvider.cs#L194

A couple ideas to help with this:

  1. Change the storage account caching logic to cache tasks instead of values, so we only allow one ValidateCredentialsAsync operation (and all subsequent GetStorageAccountAsync calls wait on that).
  2. Move storage account validation to function indexing time instead of at runtime (gets rid of a bunch of sync over async calls & blocked threads).