search

aspnet / KestrelHttpServer

[Archived] A cross platform web server for ASP.NET Core. Project moved to https://github.com/aspnet/AspNetCore
Apache License 2.0
2.63k stars 528 forks source link

An unhandled exception has occurred: Error -4077 ECONNRESET connection reset by peer #1857

Closed amirjalali1 closed 6 years ago

amirjalali1 commented 7 years ago

I have two asp.net core 1.1.2 node with an iis arr 3 as load balancer in front of it. suddenly i get the following error

An unhandled exception has occurred: Error -4077 ECONNRESET connection reset by peer

with this trace log :

Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware

System.IO.IOException: Error -4077 ECONNRESET connection reset by peer ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer

   --- End of inner exception stack trace ---

   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInput.CheckConnectionError()

   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInputExtensions.<PeekAsyncAwaited>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()

   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.ForContentLength.<PeekAsyncAwaited>d__4.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()

   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.<ReadAsyncAwaited>d__12.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.IO.StreamReader.<ReadBufferAsync>d__102.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at System.IO.StreamReader.<ReadAsyncInternal>d__69.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.WebUtilities.FormReader.<BufferAsync>d__41.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.WebUtilities.FormReader.<ReadNextPairAsyncImpl>d__34.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.WebUtilities.FormReader.<ReadFormAsync>d__43.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Http.Features.FormFeature.<InnerReadFormAsync>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenStore.<GetRequestTokensAsync>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.<ValidateRequestAsync>d__9.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ValidateAntiforgeryTokenAuthorizationFilter.<OnAuthorizationAsync>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.<Invoke>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.<Invoke>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.<Invoke>d__2.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Session.SessionMiddleware.<Invoke>d__9.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Microsoft.AspNetCore.Session.SessionMiddleware.<Invoke>d__9.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.<Invoke>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.<Invoke>d__8.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()

i also checked iis arr sc-substatus and sc-status which are 604 and 404 respectively but i don't know if its related to above issue or not.

also dump log for threads looks something like this

000000e6d97fec10 00007ff904233acf KERNELBASE!WaitForSingleObjectEx + 0x8f, calling ntdll!NtWaitForSingleObject
000000e6d97fecb0 00007ff8deed3ad3 coreclr!CLRSemaphore::Wait + 0x63, calling kernel32!WaitForSingleObjectEx
000000e6d97fece0 00007ff8dee514b5 coreclr!ThreadpoolMgr::AdjustMaxWorkersActive + 0xa5, calling coreclr!HillClimbing::Update
000000e6d97fecf0 00007ff8dedf30fa coreclr!Thread::InternalReset + 0xc6, calling coreclr!Thread::SetBackground
000000e6d97fed70 00007ff8dee037ea coreclr!ThreadpoolMgr::UnfairSemaphore::Wait + 0x126, calling coreclr!CLRSemaphore::Wait
000000e6d97fedc0 00007ff8dedf5f0f coreclr!ThreadpoolMgr::WorkerThreadStart + 0x13f, calling coreclr!ThreadpoolMgr::UnfairSemaphore::Wait
000000e6d97fedf0 00007ff8dee749f9 coreclr!EEHeapFreeInProcessHeap + 0x29, calling kernel32!HeapFreeStub
000000e6d97fee20 00007ff8dede7beb coreclr!operator delete + 0x33
000000e6d97fee60 00007ff8dedf6efb coreclr!Thread::intermediateThreadProc + 0x8b
000000e6d97feef0 00007ff90422e39a KERNELBASE!ConsoleCallServerGeneric + 0x116, calling KERNELBASE!_security_check_cookie
000000e6d97ff030 00007ff9071a8f98 ntdll!RtlpLowFragHeapAllocFromContext + 0x378, calling ntdll!memset
000000e6d97ff040 00007ff9071a8f98 ntdll!RtlpLowFragHeapAllocFromContext + 0x378, calling ntdll!memset
000000e6d97ff050 00007ff90422e178 KERNELBASE!ConsoleCallServer + 0x3c, calling KERNELBASE!ConsoleCallServerGeneric
000000e6d97ff060 00007ff9071a8f98 ntdll!RtlpLowFragHeapAllocFromContext + 0x378, calling ntdll!memset
000000e6d97ff100 00007ff9071a7a68 ntdll!RtlpAllocateHeapInternal + 0xf8, calling ntdll!RtlpLowFragHeapAllocFromContext
000000e6d97ff1c0 00007ff8edd38988 libuv!__crt_seh_guarded_call<void>::operator()<<lambda_fb3a7dec4e47f37f22dae91bb15c9095>,<lambda_698284760c8add0bfb0756c19673e34b> & __ptr64,<lambda_dfb8eca1e75fef3034a8fb18dd509707> > + 0x284, calling libuv!_acrt_add_locale_ref
000000e6d97ff1f0 00007ff8edd386b6 libuv!__crt_seh_guarded_call<void>::operator()<<lambda_5e887d1dcbef67a5eb4283622ba103bf>,<lambda_4466841279450cc726390878d4a41900> & __ptr64,<lambda_341c25c0346d94847f1f3c463c57e077> > + 0x3a, calling libuv!_acrt_unlock
000000e6d97ff200 00007ff8fb4c2355 version!DllMain + 0x35, calling kernel32!TlsSetValueStub
000000e6d97ff230 00007ff9071d0800 ntdll!RtlSetLastWin32Error + 0x40, calling ntdll!_security_check_cookie
000000e6d97ff250 00007ff8ebc8837a Microsoft_DiaSymReader_Native_amd64!_vcrt_FlsSetValue + 0x4a
000000e6d97ff280 00007ff8ebc87bd9 Microsoft_DiaSymReader_Native_amd64!_vcrt_getptd_noexit + 0xa5, calling kernel32!SetLastErrorStub
000000e6d97ff2b0 00007ff8ebc87409 Microsoft_DiaSymReader_Native_amd64!_vcrt_thread_attach + 0x9, calling Microsoft_DiaSymReader_Native_amd64!_vcrt_getptd_noexit
000000e6d97ff2e0 00007ff8ebc84b36 Microsoft_DiaSymReader_Native_amd64!_scrt_dllmain_crt_thread_attach + 0x16, calling Microsoft_DiaSymReader_Native_amd64!TMEQTS::IsTMEQTS
000000e6d97ff2f0 00007ff90719a296 ntdll!RtlDeactivateActivationContextUnsafeFast + 0xf6, calling ntdll!_security_check_cookie
000000e6d97ff310 00007ff8ebc84559 Microsoft_DiaSymReader_Native_amd64!dllmain_crt_dispatch + 0x2d, calling Microsoft_DiaSymReader_Native_amd64!_scrt_dllmain_crt_thread_attach
000000e6d97ff320 00007ff907197e53 ntdll!RtlActivateActivationContextUnsafeFast + 0x93, calling ntdll!_security_check_cookie
000000e6d97ff340 00007ff8ebc847a0 Microsoft_DiaSymReader_Native_amd64!dllmain_dispatch + 0x74, calling Microsoft_DiaSymReader_Native_amd64!DllMain
000000e6d97ff380 00007ff907177874 ntdll!LdrpCallTlsInitializers + 0x3c, calling ntdll!RtlReleaseSRWLockShared
000000e6d97ff3a0 00007ff90719a35f ntdll!LdrpCallInitRoutine + 0x4b
000000e6d97ff3c0 00007ff90717fd30 ntdll!LdrpReleaseLoaderLock + 0x20, calling ntdll!RtlLeaveCriticalSection
000000e6d97ff3d0 00007ff9071ebab4 ntdll!LdrpDropLastInProgressCount + 0x38, calling ntdll!RtlLeaveCriticalSection
000000e6d97ff400 00007ff907197d6a ntdll!LdrpInitializeThread + 0x20a, calling ntdll!LdrpDropLastInProgressCount
000000e6d97ff470 00007ff907197cac ntdll!LdrpInitializeThread + 0x14c, calling ntdll!RtlActivateActivationContextUnsafeFast
000000e6d97ff478 00007ff907197cdb ntdll!LdrpInitializeThread + 0x17b, calling ntdll!RtlDeactivateActivationContextUnsafeFast
000000e6d97ff4c0 00007ff907197d35 ntdll!LdrpInitializeThread + 0x1d5, calling ntdll!RtlActivateActivationContextUnsafeFast
000000e6d97ff4c8 00007ff907197d52 ntdll!LdrpInitializeThread + 0x1f2, calling ntdll!RtlDeactivateActivationContextUnsafeFast
000000e6d97ff4f0 00007ff9071ea092 ntdll!_LdrpInitialize + 0x8e, calling ntdll!NtTestAlert
000000e6d97ff570 00007ff9071e9fb8 ntdll!LdrInitializeThunk + 0x18, calling ntdll!NtContinue
000000e6d97ff9e0 00007ff8dedf6ed6 coreclr!Thread::intermediateThreadProc + 0x66, calling coreclr!_chkstk
000000e6d97ffa20 00007ff906a58364 kernel32!BaseThreadInitThunk + 0x14, calling kernel32!guard_dispatch_icall_nop
000000e6d97ffa50 00007ff9071d70d1 ntdll!RtlUserThreadStart + 0x21, calling ntdll!guard_dispatch_icall_nop

but again i don't know if dump is related to error -4077 or not

any suggestion or help would be appreciated?

halter73 commented 7 years ago

This exception indicates that the client aborted a request in the middle of sending a request body. Other than the logs, are you seeing any unexpected behavior?

amirjalali1 commented 7 years ago

yes after this error i got 502.3 error then i have to recycle application pool

fugaku commented 7 years ago

I also got this issue when the client tries to upload a file

halter73 commented 7 years ago

@1amirjalai @fugaku After either of you see the -4077 ECONNRESET in your logs, is that the last thing logged? Are future requests not processed or logged? Is this the reason you have to restart the server?

If either of you could provide an app that reproduces the issue with either a client app included or instructions on how to make a request that causes the server to become unresponsive that would help.

If the server continues processing future requests, then this might just be the expected behavior when a client disconnects in the middle of an upload.

fugaku commented 7 years ago

Subsequent requests are processed. Error will occur again if the user tries to upload another file.

halter73 commented 7 years ago

@fugaku Do that mean that all file uploads fail? Can you provide an app that reproduces this behavior?

muratg commented 7 years ago

@1amirjalai what about @halter73's question?