SslStream object disposed exception

timmydo commented 5 years ago

I'm not sure if this is the right place to report this, but I'm seeing this exception occasionally:

kubernetes docker pod uname -a: Linux platform-prod-577cf48b4b-wrg7x 4.15.0-1030-azure #31~16.04.1-Ubuntu SMP Tue Oct 30

context: running kestrel behind nginx on linux in docker image based on dotnet:2.1-aspnetcore-runtime

19:40:01 UTC 2018 x86_64 GNU/Linux
Unhandled Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

pod error code:

Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Fri, 07 Dec 2018 17:06:46 -0800
      Finished:     Fri, 07 Dec 2018 23:48:04 -0800

let me know if there is more info I could provide. thanks.

halter73 commented 5 years ago

System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)

The ODE seems to have happened during TLS renegotiation. Kestrel ensures that it's done with its SslStreams before disposing them meaning that any calls to AuthenticateAsServerAsync(), ReadAsync(), WriteAsync(), etc... have completed.

@timmydo Do you have any evidence that the SslStream belongs to Kestrel? Kestrel doesn't initiate a renegotiations itself, and I don't know why a client would do so either. I suspect that HttpClient might own the SslStream.

Either way, we should probably move this issue to dotnet/corefx.

halter73 commented 5 years ago

This issue was moved to dotnet/corefx#34033

aspnet / KestrelHttpServer

SslStream object disposed exception #3111