search

aspnet / HttpSysServer

[Archived] A web server for ASP.NET Core based on the Windows Http Server API. Project moved to https://github.com/aspnet/AspNetCore
Apache License 2.0
106 stars 39 forks source link

The Http.Sys server has shut down. And an issue that hangs when I run it again. #489

Closed wxilas21 closed 5 years ago

wxilas21 commented 5 years ago

There is an issue that hangs when running HttpSys Server for more than one day, shutting down and restarting. And doing TaskKill in this state does not kill the process. In this case, there are issues that require a reboot. We have a dump file at this time. I would appreciate your help in solving the problem.

Execution environment

Using AWS EC2 Windows server 2012 R2 Version 6.3 ( build 9600) I used Dotnet core SDK 2.1.403, Runtime 2.1.5

Tratcher commented 5 years ago

Is there a reason you're using Http.Sys and Kestrel? (https://github.com/aspnet/KestrelHttpServer/issues/3074)

wxilas21 commented 5 years ago

The reason I started using Kestrel Server was that I used the Dotnet framework for the server of the previous service. Also, since the next project should be able to service in Linux, I have to use dotnet core which supports cross platform.

I started the project three months ago using Kestrel. This (https://github.com/aspnet/KestrelHttpServer/issues/3074) has encountered a problem and I was considering using HttpSys. Because there was an internal opinion that the server should run on Windows. However, I found another problem with HttpSys. And I have done many tests. But I can not find a solution and I ask one question.

Please advise on this issue.

Tratcher commented 5 years ago

Open the dump in Visual Studio or WinDbg and look at the call stacks for active threads. How many threads do you see? Are any stuck at suspicious places like calls to Wait()?

wxilas21 commented 5 years ago

I checked the dump. The thread only had one main thread. This is the call stack of the main thread.

Main Thread Call Site

ntdll!NtDeviceIoControlFile+0xa httpapi!HttpApiSynchronousDeviceControl+0x73 httpapi!HttpCreateServerSession+0x6e 0x00007fff35ddad2d Microsoft_AspNetCore_Server_HttpSys!Microsoft.AspNetCore.Server.HttpSys.ServerSession..ctor()$##60001E3+0x47 Microsoft_AspNetCore_Server_HttpSys!Microsoft.AspNetCore.Server.HttpSys.HttpSysListener..ctor(Microsoft.AspNetCore.Server.HttpSys.HttpSysOptions, Microsoft.Extensions.Logging.ILoggerFactory)$##6000171+0xd1 Microsoft_AspNetCore_Server_HttpSys!Microsoft.AspNetCore.Server.HttpSys.MessagePump..ctor(Microsoft.Extensions.Options.IOptions1, Microsoft.Extensions.Logging.ILoggerFactory, Microsoft.AspNetCore.Authentication.IAuthenticationSchemeProvider)$##60001A2+0x84 coreclr!CallDescrWorkerInternal+0x83 [E:\A_work\36\s\src\vm\amd64\CallDescrWorkerAMD64.asm @ 101] coreclr!CallDescrWorkerReflectionWrapper+0x1a [e:\a_work\36\s\src\vm\reflectioninvocation.cpp @ 740] coreclr!RuntimeMethodHandle::InvokeMethod+0x4bb [e:\a_work\36\s\src\vm\reflectioninvocation.cpp @ 1355] System_Private_CoreLib!System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)$##6003480+0xbc System_Private_CoreLib!System.Reflection.ConstructorInfo.Invoke(System.Object[])$##6003276+0x1f Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)$##6000051+0x9b Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator..ctor()$##6000067+0xf2 Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)$##6000053+0x77 Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitSingleton(Microsoft.Extensions.DependencyInjection.ServiceLookup.SingletonCallSite, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)$##6000052+0x1c Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator..ctor()$##6000067+0x146 Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine+<>cDisplayClass1_0.b0(Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)$##6000129+0x77 Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(System.Type, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)$##60000C7+0x67 Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(System.Type)$##60000C4+0xf Microsoft_Extensions_DependencyInjection!Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(System.Type)$##600002E+0x18 Microsoft_Extensions_DependencyInjection_Abstractions!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(System.IServiceProvider, System.Type)$##6000040+0x56 Microsoft_Extensions_DependencyInjection_Abstractions!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(System.IServiceProvider, System.Type)$##6000040+0x121 Microsoft_AspNetCore_Hosting!Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureServer()$##60000D6+0x27 0x00007fff`35dd9cd7 Microsoft_AspNetCore_Hosting!Microsoft.AspNetCore.Hosting.Internal.WebHost+d26.MoveNext()$##60001F5+0xa4 0x00007fff35dd9089 0x00007fff35dd9010 Microsoft_AspNetCore_Hosting!Microsoft.AspNetCore.Hosting.Internal.WebHost.Start()$##60000D1+0x1c 0x00007fff`35da1505 coreclr!CallDescrWorkerInternal+0x83 [E:\A_work\36\s\src\vm\amd64\CallDescrWorkerAMD64.asm @ 101] coreclr!CallDescrWorkerWithHandler+0x53 [e:\a_work\36\s\src\vm\callhelpers.cpp @ 78] coreclr!MethodDescCallSite::CallTargetWorker+0x2b5 [e:\a_work\36\s\src\vm\callhelpers.cpp @ 628] coreclr!MethodDescCallSite::Call+0xb [e:\a_work\36\s\src\vm\callhelpers.h @ 433] coreclr!RunMain+0x229 [e:\a_work\36\s\src\vm\assembly.cpp @ 1707] coreclr!Assembly::ExecuteMainMethod+0x228 [e:\a_work\36\s\src\vm\assembly.cpp @ 1817] coreclr!CorHost2::ExecuteAssembly+0x206 [e:\a_work\36\s\src\vm\corhost.cpp @ 491] coreclr!coreclr_execute_assembly+0xd6 [e:\a_work\36\s\src\dlls\mscoree\unixinterface.cpp @ 407] hostpolicy!run+0xfd0 hostpolicy!corehost_main+0x7d hostfxr!execute_app+0x1d8 hostfxr!fx_muxer_t::read_config_and_execute+0xb7a hostfxr!fx_muxer_t::handle_exec_host_command+0x242 hostfxr!fx_muxer_t::execute+0x346 hostfxr!hostfxr_main_startupinfo+0x95 GameServer_exe!run+0x370 GameServer_exe!invoke_main+0x22 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 79] GameServer_exe!scrt_common_main_seh+0x11d [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253] kernel32!BaseThreadInitThunk+0x22 ntdll!RtlUserThreadStart+0x34

Tratcher commented 5 years ago

That call stack is the server trying to start up. I don't know why it would hang, I'll ask some httpapi experts.

In the meantime, can you clarify your initial statement? You had successfully started your application and served traffic for a day before it hung. That's when you shut it down and restarted it and ran into the issue with the stack trace given above. Is this correct? Do you have any dumps from the initial issue?

Tratcher commented 5 years ago

Interesting link.

wxilas21 commented 5 years ago

That call stack is the server trying to start up. I don't know why it would hang, I'll ask some httpapi experts.

In the meantime, can you clarify your initial statement? You had successfully started your application and served traffic for a day before it hung. That's when you shut it down and restarted it and ran into the issue with the stack trace given above. Is this correct? Do you have any dumps from the initial issue?

Yes, that's right. Symptoms appeared after a day or so. And when you restart, it hangs. And this Call Site is a dump when it hangs.

We'll look at the link you've shown and we'll look at it in detail.

wxilas21 commented 5 years ago

I saw the link you gave me. It is similar to my situation. When the HttpSys server shut down, the console identified the remaining issues in the TaskManager that disappeared. And when you start again, the program hangs in the same situation as the dump.

Tratcher commented 5 years ago

Can you capture a dump before restarting the process? It will be interesting to see what the original error was.

Tratcher commented 5 years ago

Sharing the other dump may also be interesting. You can send a download link to the e-mail address in my profile.

wxilas21 commented 5 years ago

The next update is (UTC + 9) 11-09 AM 10:30. I will get a dump and send you an email before updating. Thank you.

wxilas21 commented 5 years ago

I sent an email. Thank you.

shirhatti commented 5 years ago

As I mentioned in https://github.com/aspnet/KestrelHttpServer/issues/3074#issuecomment-439545577, try installing the KB patch. I've verified that the TCP/IP bug in Windows was causing that issue.

wxilas21 commented 5 years ago

Thank you for your interest in the old issue. I will check if Windows KB4338824 is updated. And @Tratcher continues to help you solve this issue.

I will leave a note as soon as I can confirm that this update is installed or not.

Thank you.

shirhatti commented 5 years ago

We're closing this issue as no response or updates have been provided in a timely manner and we have been unable to reproduce it. If you have more details and are encountering this issue please add a new reply and re-open the issue.