Remote Host Closed the Connection Exception - rock-rt

[mikedotmundy]

Description

We are seeing a new instance of the exception The remote host closed the connection. The Error code is 0x80070057. I believe this has something to do with the RealTime Engine.

I was able recreate this on the demo site via the Obsidian Attendance Detail Block. I am not sure if other blocks also cause this issue. See steps below for how to create the exception.

Even though I confirmed this issue in v15.1 and v16.0, I believe this exception began in v15.0.

Stack Trace:

at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data) in /_/src/Microsoft.AspNet.SignalR.Core/Owin/ServerResponse.cs:line 47
   at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data) in /_/src/Microsoft.AspNet.SignalR.Core/Hosting/ResponseExtensions.cs:line 32
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context) in /_/src/Microsoft.AspNet.SignalR.Core/PersistentConnection.cs:line 513
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context) in /_/src/Microsoft.AspNet.SignalR.Core/PersistentConnection.cs:line 205
   at Rock.RealTime.AspNet.AspNetEngineStartup.RegisterSignalRClaims(IOwinContext context, Func`1 nextHandler) in \Rock\RealTime\AspNet\AspNetEngineStartup.cs:line 140
   at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Actual Behavior

An exception is generated.

Expected Behavior

No exception.

Steps to Reproduce

• Go to https://rockrmslatest.com/page/364?GroupId=110&OccurrenceId=834 or pick a different Group and AttendanceOccurrence.
• Click on Noah Decker to add an attendance record.
• View exception log to see there is no exception.
• Click on Noah Decker again to remove his attendance record.
• View exception log and see there is now an exception.

From what I could tell, this only occurs when removing an attendance record, not when adding.

Issue Confirmation

• [X] Perform a search on the Github Issues to see if your bug or enhancement is already reported.
• [X] Try to reproduce the problem on a fresh install or on the demo site.

Rock Version

v15.1, v16.0

Client Culture Setting

en-US

[joshuahenninger]

@mikedotmundy This is fixed in v16.1 in this commit.

This issue was easily reproducible on Pre-Alpha following the steps you provided. However, we had a difficult time reproducing it locally on our end, even when using IIS (not IIS Express). That said, hooking into the SignalR pipeline and ignoring this specific exception may be the best bet to prevent these exceptions from being logged.

If this issue persists after the next Pre-Alpha deployment, we can re-open and try another approach. Please let me know if you have any questions or concerns.

[JimMichael]

We are still seeing lots of these in 1.16.1.1 alpha, all with GetFile requests to avatars.

I'm unclear from the comment above if this WILL be (attempted to be) fixed in 16.1 when it releases and is just NOT in the 16.1 alpha we're currently testing, or if it will be fixed in a post 16.1 future release.

[nairdo]

We are still seeing lots of these in 1.16.1.1 alpha, all with GetFile requests to avatars.

I'm unclear from the comment above if this WILL be (attempted to be) fixed in 16.1 when it releases and is just NOT in the 16.1 alpha we're currently testing, or if it will be fixed in a post 16.1 future release.

The fix for this issue is for Rock RealTime. But you might be referring to the fix in v17 for issue #5521? If so I can look into trying to move that ahead of schedule and try to get it into v16.2, but it will depend on a few factors including risk.

[JimMichael]

The fix for this issue is for Rock RealTime. But you might be referring to the fix in v17 for issue #5521? If so I can look into trying to move that ahead of schedule and try to get it into v16.2, but it will depend on a few factors including risk.

Thanks @nairdo, I was indeed mixing up two issues. We're in no rush to get this fixed, they cause no harm other than making the exception log ugly ;-) I was just trying to get clarification on when the "attempted fix" would arrive, so thanks for that!

[joshuahenninger]

@mikedotmundy We were able to reproduce this issue in Pre-Alpha after the previous fix (in v16.1), and have since added more changes to suppress these log messages in v16.3 in this commit.

The latest changes seem to have fixed the issue as Pre-Alpha is no longer logging these messages from the Rock RealTime system, and we've updated this issue with the "Fixed in v16.3" label. Please let me know if you have any questions.

[mikedotmundy]

We are still seeing this in v16.3, however, it is much less frequent. We have only seen it occur once in the just over 24 hours since being on v16.3.

[chead4]

@mikedotmundy Thanks for the update Mike. Please provide the stack trace(s) and we will include these log messages to suppress.

[mikedotmundy]

 at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
   at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
   at Rock.RealTime.AspNet.AspNetEngineStartup.RegisterSignalRClaims(IOwinContext context, Func`1 nextHandler) in \Rock\RealTime\AspNet\AspNetEngineStartup.cs:line 153
   at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[mikedotmundy]

Here is one that I received this morning that is slightly different:

at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
   at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
   at Rock.RealTime.AspNet.AspNetEngineStartup.RegisterSignalRClaims(IOwinContext context, Func`1 nextHandler) in \Rock\RealTime\AspNet\AspNetEngineStartup.cs:line 153
   at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Mapping.MapMiddleware.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[nairdo]

Thanks @mikedotmundy We have an idea how these are sneaking by our traps. We'll include additional checks in v16.4 for these.