aspnet / AspNetKatana

Microsoft's OWIN implementation, the Katana project
Apache License 2.0
966 stars 334 forks source link

Page crashes related to Owin #546

Open nithyaRagupathy opened 4 days ago

nithyaRagupathy commented 4 days ago

We are recently seeing multiple page crashes related to Owin and need SME help to investigate the issue.

Stack trace

We are seeing multiple page crashes related to Owin and need SME help to investigate the issue.

In our Code, we are using the packages with these versions- Microsoft.Owin - 4.2.2 "Microsoft.Identity.ServiceEssentials.Extensions.OwinMiddleware - "1.28.1" Microsoft.Identity.ServiceEssentials.Extensions.OwinMiddleware.S2S - "1.28.1" Microsoft.IdentityModel.S2S" - "4.10.0" Microsoft.IdentityModel.S2S.Tokens" - "4.10.0" Microsoft.IdentityModel.S2S.Configuration" - "4.10.0" Microsoft.IdentityModel.S2S.Extensions.Owin" - "4.9.1" Microsoft.IdentityModel.S2S.Extensions.AspNetCore" - "4.10.0" Microsoft.IdentityModel.Abstractions" - "8.1.0" Microsoft.IdentityModel.JsonWebTokens" - "8.1.0" Microsoft.IdentityModel.Logging" - "8.1.0" Microsoft.IdentityModel.Protocols" - "8.1.0" Microsoft.IdentityModel.Protocols.OpenIdConnect" - "8.1.0" Microsoft.IdentityModel.Protocols.SignedHttpRequest" - "8.1.0" Microsoft.IdentityModel.Protocols.WsFederation" - "8.1.0" Microsoft.IdentityModel.Tokens" - "8.1.0" Microsoft.IdentityModel.Validators" - "8.1.0" Microsoft.IdentityModel.Xml" - "8.1.0" System.IdentityModel.Tokens.Jwt" - "8.1.0" Microsoft.Identity.Abstractions" - "6.0.0"

Given below is the stack trace from Watson dump

msvcrt!ascii_stricmp+0xd [minkernel\crts\crtw32\string\stricmp.c @ 124] iiscore!W3_RESPONSE::SetHeader+0x458 [servercommon\inetsrv\iis\iisrearc\iis70\core\w3response.cxx @ 638] webengine4!MgdSetUnknownHeader+0xa7 [f:\dd\ndp\fx\src\xsp\webengine\mgdexports.cxx @ 854] system_web!DomainNeutralILStubClass.IL_STUB_PInvoke+0xd6 system_web!System.Web.Hosting.IIS7WorkerRequest.SetUnknownResponseHeader+0xa7 [f:\dd\ndp\fx\src\xsp\system\Web\Hosting\IIS7WorkerRequest.cs @ 2076] system_web!System.Web.Hosting.IIS7WorkerRequest.SetKnownResponseHeader+0x70 [f:\dd\ndp\fx\src\xsp\system\Web\Hosting\IIS7WorkerRequest.cs @ 2063] system_web!System.Web.HttpHeaderCollection.SetHeader+0x147 [f:\dd\ndp\fx\src\xsp\system\Web\HttpHeaderCollection.cs @ 129] microsoft_owin_hostsystemweb!Microsoft.Owin.Host.SystemWeb.CallHeaders.AspNetResponseHeaders.Set+0xc9 [//src/Microsoft.Owin.Host.SystemWeb/CallHeaders/AspNetResponseHeaders.cs @ 108] microsoftowin!Microsoft.Owin.ResponseCookieCollection.Append+0x35e [//src/Microsoft.Owin/ResponseCookieCollection.cs @ 76] microsoft_owin_securityopenidconnect!Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler.RememberNonce+0x1a7 [//src/Microsoft.Owin.Security.OpenIdConnect/OpenidConnectAuthenticationHandler.cs @ 676] microsoft_owin_security_openidconnect!Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler+_ApplyResponseChallengeAsync_d_10 [//src/Microsoft.Owin.Security.OpenIdConnect/OpenidConnectAuthenticationHandler.cs @ 195] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39 [[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39, mscorlib @ 322] microsoft_owin_securityopenidconnect!Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler.ApplyResponseChallengeAsync+0x4e [//src/Microsoft.Owin.Security.OpenIdConnect/OpenidConnectAuthenticationHandler.cs @ 640] microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler+_ApplyResponseCoreAsync_d40 [/_/src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 179] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39 [[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39, mscorlib @ 322] microsoft_owinsecurity!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.ApplyResponseCoreAsync+0x4e [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 207] mscorlib!System.Threading.LazyInitializer.EnsureInitializedCore[[System.Canon, mscorlib]]+0x8b [[System.Canon, mscorlib @ 241] microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler+_ApplyResponseAsync_d39 [/_/src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 157] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39 [[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39, mscorlib @ 322] microsoft_owinsecurity!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.ApplyResponseAsync+0x4e [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 207] microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler+_TeardownAsync_d34 [/_/src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 95] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39 [[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d39, mscorlib @ 322] microsoft_owinsecurity!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.TeardownAsync+0x4e [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 128] microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware1+_Invoke_d__5 [[System.__Canon, mscorlib @ 32] mscorlib!System.Threading.ExecutionContext.RunInternal+0x172 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 980] mscorlib!System.Threading.ExecutionContext.Run+0x15 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 928] mscorlib!System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\AsyncMethodBuilder.cs @ 1070] mscorlib!System.Threading.Tasks.AwaitTaskContinuation.RunCallback+0x6a [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\TaskContinuation.cs @ 759] mscorlib!System.Threading.Tasks.Task.FinishContinuations+0xfe [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 3642] mscorlib!System.Threading.Tasks.Task1[[System.Canon, mscorlib]].TrySetResult+0x9ad45 [[System.Canon, mscorlib @ 463] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder1[[System.__Canon, mscorlib]].SetResult+0x0 [[System.__Canon, mscorlib @ 590] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder1[[System.Canon, mscorlib]].SetResult+0x0 [[System.Canon, mscorlib @ 638] microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+_Invoke_d5 [[System.Canon, mscorlib @ 30] mscorlib!System.Threading.ExecutionContext.RunInternal+0x172 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 980] mscorlib!System.Threading.ExecutionContext.Run+0x15 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 928] mscorlib!System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\AsyncMethodBuilder.cs @ 1070] system_web!System.Web.Util.SynchronizationHelper.SafeWrapCallback+0x76 [f:\dd\ndp\fx\src\xsp\system\Web\Util\SynchronizationHelper.cs @ 159] mscorlib!System.Threading.Tasks.Task.Execute+0x47 [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2498] mscorlib!System.Threading.ExecutionContext.RunInternal+0x172 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 980] mscorlib!System.Threading.ExecutionContext.Run+0x15 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 928] mscorlib!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0x231 [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2827] mscorlib!System.Threading.Tasks.Task.ExecuteEntry+0xa1 [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2767] mscorlib!System.Threading.ThreadPoolWorkQueue.Dispatch+0x156 [f:\dd\ndp\clr\src\BCL\system\threading\threadpool.cs @ 820] clr!CallDescrWorkerInternal+0x83 clr!CallDescrWorkerWithHandler+0x4e clr!MethodDescCallSite::CallTargetWorker+0xfa clr!QueueUserWorkItemManagedCallback+0x2a clr!ManagedThreadBase_DispatchInner+0x40 clr!ManagedThreadBase_DispatchMiddle+0x6c clr!ManagedThreadBase_DispatchOuter+0x4c clr!ManagedThreadBase_DispatchInCorrectAD+0x15 clr!Thread::DoADCallBack+0x26b clr!ManagedThreadBase_DispatchInner+0x2e57 clr!ManagedThreadBase_DispatchMiddle+0x6c clr!ManagedThreadBase_DispatchOuter+0x4c clr!ManagedThreadBase_FullTransitionWithAD+0x2f clr!ManagedPerAppDomainTPCount::DispatchWorkItem+0xa4 clr!ThreadpoolMgr::ExecuteWorkRequest+0x64 clr!ThreadpoolMgr::WorkerThreadStart+0xf6 clr!Thread::intermediateThreadProc+0x8b kernel32!BaseThreadInitThunk+0x10 [clientcore\base\win32\client\thread.c @ 75] ntdll!RtlUserThreadStart+0x2b [minkernel\ntdll\rtlstrt.c @ 1152]

halter73 commented 4 days ago

Upgrading package versions seems like a plausible cause for issues. We’ve seen this a lot of issues with the Midrosoft.IdentityModel.* package versions getting misaligned.

I notice that the list of NuGet packages that you’ve included in the issue do not include some transitive dependencies. For example, I see the Microsoft.Owin.Security.OpenIdConnect in the stack trace, but I don’t see the package version for it. Using the latest dotnet SDKs, you can run something like the following to get all the transitive packages:

$ dotnet list package --include-transitive
Project 'OwinOidc' has the following package references
   Top-level Package                                      Requested    Resolved
   > Microsoft.AspNet.Mvc                                 5.2.9        5.2.9
   > Microsoft.AspNet.WebPages                            3.2.9        3.2.9
   > Microsoft.IdentityModel.Logging                      7.0.2        7.0.2
   > Microsoft.IdentityModel.Protocols                    7.0.2        7.0.2
   > Microsoft.IdentityModel.Protocols.OpenIdConnect      7.0.2        7.0.2
   > Microsoft.IdentityModel.Tokens                       7.0.2        7.0.2
   > Microsoft.Owin                                       4.2.2        4.2.2
   > Microsoft.Owin.Host.SystemWeb                        4.2.2        4.2.2
   > Microsoft.Owin.Hosting                               4.2.2        4.2.2
   > Microsoft.Owin.Security                              4.2.2        4.2.2
   > Microsoft.Owin.Security.Cookies                      4.2.2        4.2.2
   > Microsoft.Owin.Security.OpenIdConnect                4.2.2        4.2.2
   > Microsoft.Owin.Security.WsFederation                 4.2.2        4.2.2
   > Newtonsoft.Json                                                   3.5.8
   > Owin                                                 1.0.0        1.0.0
   > System.Buffers                                       4.5.1        4.5.1
   > System.IdentityModel.Tokens.Jwt                      7.0.2        7.0.2
   > System.Management.Automation.dll                     10.0.10586   10.0.10586
   > System.Numerics.Vectors                              4.5.0        4.5.0
   > System.Runtime.CompilerServices.Unsafe               6.0.0        6.0.0

   Transitive Package                                    Resolved
   > Microsoft.AspNet.Razor                              3.2.9
   > Microsoft.IdentityModel.Abstractions                7.0.2
   > Microsoft.IdentityModel.JsonWebTokens               7.0.2
   > Microsoft.IdentityModel.Protocols.WsFederation      5.3.0
   > Microsoft.IdentityModel.Tokens.Saml                 5.3.0
   > Microsoft.IdentityModel.Xmll                        5.3.0

This is just an example from a previous similar issue. I don’t know what feed Microsoft.Identity.ServiceEssentials.Extensions.OwinMiddleware and the like are coming from, so I don’t know what your transitive dependencies are, but mixing 7.0.2 Microsoft.IdentityModel. packages with other transitive 5.3.0 Microsoft.IdentityModel. packages caused problems in that case. I suspect you’re running into a similar issue.

Otherwise, it’s hard to diagnose what’s going on with just a stack trace. What’s happening at that stack trace? We normally need an exception type and message at a minimum. If it’s an AV, a dump might be helpful, but as far as I know, the Microsoft.Owin packages referenced in the stack trace don’t have any unsafe logic that could reasonably cause an AV, and they have not been updated in a couple years.

[1]: [Feature Request] Warn when different versions of the IdentityModel NuGet packages are used (#2513)