dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.21k stars 9.95k forks source link

w3wp.exe crashes with Exception code 0xc0000409 #6834

Closed Giorgi closed 5 years ago

Giorgi commented 5 years ago

Describe the bug

Application pool (w3wp.exe) crashes periodically with Exception code 0xc0000409.

To Reproduce

Steps to reproduce the behavior:

  1. Using this version of ASP.NET Core 2.2.0 targeting Full Framework 4.7.1
  2. Server: Windows Server 2016 (OS Build 14393.2670)
  3. ANCM version: 12.2.18316.0

I see this error in event log:

Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a Faulting module name: ucrtbase.dll, version: 10.0.14393.2636, time stamp: 0x5bda7e9c Exception code: 0xc0000409 Fault offset: 0x000000000006e83e Faulting process id: 0x1894 Faulting application start time: 0x01d4af0db64a2704 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Windows\System32\ucrtbase.dll Report Id: 45957b36-832b-430c-8269-42a6dcc5caec Faulting package full name: Faulting package-relative application ID:

Giorgi commented 5 years ago

Data from Report.wer:

Version=1 EventType=BEX64 EventTime=131922772788813152 ReportType=2 Consent=1 UploadTime=131922772821782055 ReportIdentifier=5782021c-1b03-11e9-b81a-005056b2c05b IntegratorReportIdentifier=45957b36-832b-430c-8269-42a6dcc5caec NsAppName=w3wp.exe AppSessionGuid=00001894-0000-0022-0427-4ab60dafd401 TargetAppId=W:0000f519feec486de87ed73cb92d3cac802400000000!000054443c275eddd9bbe75c6d47cf9c24076b5d785c!w3wp.exe TargetAppVer=2016//07//16:02:27:22!15ee0!w3wp.exe BootId=4294967295 Response.type=4 Sig[0].Name=Application Name Sig[0].Value=w3wp.exe Sig[1].Name=Application Version Sig[1].Value=10.0.14393.0 Sig[2].Name=Application Timestamp Sig[2].Value=57899b8a Sig[3].Name=Fault Module Name Sig[3].Value=ucrtbase.dll Sig[4].Name=Fault Module Version Sig[4].Value=10.0.14393.2636 Sig[5].Name=Fault Module Timestamp Sig[5].Value=5bda7e9c Sig[6].Name=Exception Offset Sig[6].Value=000000000006e83e Sig[7].Name=Exception Code Sig[7].Value=c0000409 Sig[8].Name=Exception Data Sig[8].Value=0000000000000007 DynamicSig[1].Name=OS Version DynamicSig[1].Value=10.0.14393.2.0.0.272.7 DynamicSig[2].Name=Locale ID DynamicSig[2].Value=1033 DynamicSig[22].Name=Additional Information 1 DynamicSig[22].Value=e013 DynamicSig[23].Name=Additional Information 2 DynamicSig[23].Value=e013440e2dba3ede24b3360c0f76aee5 DynamicSig[24].Name=Additional Information 3 DynamicSig[24].Value=3786 DynamicSig[25].Name=Additional Information 4 DynamicSig[25].Value=3786abc58e43f0878a893f06ca815378 UI[2]=c:\windows\system32\inetsrv\w3wp.exe UI[5]=Check online for a solution (recommended) UI[6]=Check for a solution later (recommended) UI[7]=Close UI[8]=IIS Worker Process stopped working and was closed UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available. UI[10]=&Close LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll LoadedModule[4]=C:\Windows\System32\msvcrt.dll LoadedModule[5]=C:\Windows\System32\combase.dll LoadedModule[6]=C:\Windows\System32\ucrtbase.dll LoadedModule[7]=C:\Windows\System32\RPCRT4.dll LoadedModule[8]=C:\Windows\System32\bcryptPrimitives.dll LoadedModule[9]=c:\windows\system32\inetsrv\iisutil.dll LoadedModule[10]=C:\Windows\System32\advapi32.dll LoadedModule[11]=C:\Windows\System32\sechost.dll LoadedModule[12]=C:\Windows\System32\WS2_32.dll LoadedModule[13]=C:\Windows\System32\kernel.appcore.dll LoadedModule[14]=C:\Windows\SYSTEM32\ntmarta.dll LoadedModule[15]=C:\Windows\System32\user32.dll LoadedModule[16]=C:\Windows\System32\win32u.dll LoadedModule[17]=C:\Windows\System32\GDI32.dll LoadedModule[18]=C:\Windows\System32\gdi32full.dll LoadedModule[19]=c:\windows\system32\inetsrv\w3wphost.dll LoadedModule[20]=C:\Windows\System32\OLEAUT32.dll LoadedModule[21]=C:\Windows\System32\msvcp_win.dll LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll LoadedModule[23]=C:\Windows\SYSTEM32\XmlLite.dll LoadedModule[24]=C:\Windows\SYSTEM32\bcrypt.dll LoadedModule[25]=C:\Windows\SYSTEM32\ncrypt.dll LoadedModule[26]=C:\Windows\SYSTEM32\ktmw32.dll LoadedModule[27]=C:\Windows\SYSTEM32\NTASN1.dll LoadedModule[28]=c:\windows\system32\inetsrv\IISRES.DLL LoadedModule[29]=C:\Windows\SYSTEM32\CRYPTSP.dll LoadedModule[30]=C:\Windows\system32\rsaenh.dll LoadedModule[31]=C:\Windows\SYSTEM32\CRYPTBASE.dll LoadedModule[32]=C:\Windows\System32\clbcatq.dll LoadedModule[33]=C:\Windows\system32\mlang.dll LoadedModule[34]=C:\Windows\system32\inetsrv\iiscore.dll LoadedModule[35]=c:\windows\system32\inetsrv\w3dt.dll LoadedModule[36]=c:\windows\system32\inetsrv\W3TP.dll LoadedModule[37]=C:\Windows\SYSTEM32\SspiCli.dll LoadedModule[38]=C:\Windows\SYSTEM32\HTTPAPI.dll LoadedModule[39]=C:\Windows\system32\napinsp.dll LoadedModule[40]=C:\Windows\System32\mswsock.dll LoadedModule[41]=C:\Windows\SYSTEM32\DNSAPI.dll LoadedModule[42]=C:\Windows\System32\NSI.dll LoadedModule[43]=C:\Windows\SYSTEM32\IPHLPAPI.DLL LoadedModule[44]=C:\Windows\System32\winrnr.dll LoadedModule[45]=C:\Windows\system32\NLAapi.dll LoadedModule[46]=C:\Windows\System32\rasadhlp.dll LoadedModule[47]=C:\Windows\System32\fwpuclnt.dll LoadedModule[48]=C:\Windows\System32\inetsrv\loghttp.dll LoadedModule[49]=C:\Windows\System32\inetsrv\cachuri.dll LoadedModule[50]=C:\Windows\System32\inetsrv\cachfile.dll LoadedModule[51]=C:\Windows\System32\inetsrv\cachtokn.dll LoadedModule[52]=C:\Windows\System32\inetsrv\cachhttp.dll LoadedModule[53]=C:\Windows\System32\inetsrv\compstat.dll LoadedModule[54]=C:\Windows\System32\inetsrv\defdoc.dll LoadedModule[55]=C:\Windows\System32\inetsrv\dirlist.dll LoadedModule[56]=C:\Windows\System32\inetsrv\protsup.dll LoadedModule[57]=C:\Windows\System32\inetsrv\static.dll LoadedModule[58]=C:\Windows\System32\inetsrv\authanon.dll LoadedModule[59]=C:\Windows\System32\inetsrv\modrqflt.dll LoadedModule[60]=C:\Windows\System32\inetsrv\custerr.dll LoadedModule[61]=C:\Windows\System32\inetsrv\isapi.dll LoadedModule[62]=C:\Windows\System32\inetsrv\filter.dll LoadedModule[63]=C:\Windows\System32\inetsrv\validcfg.dll LoadedModule[64]=C:\Windows\system32\inetsrv\aspnetcore.dll LoadedModule[65]=C:\Windows\SYSTEM32\WINHTTP.dll LoadedModule[66]=C:\Windows\SYSTEM32\ondemandconnroutehelper.dll LoadedModule[67]=C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll LoadedModule[68]=C:\Windows\System32\SHELL32.dll LoadedModule[69]=C:\Windows\System32\cfgmgr32.dll LoadedModule[70]=C:\Windows\System32\windows.storage.dll LoadedModule[71]=C:\Windows\System32\powrprof.dll LoadedModule[72]=C:\Windows\System32\shlwapi.dll LoadedModule[73]=C:\Windows\System32\shcore.dll LoadedModule[74]=C:\Windows\System32\profapi.dll LoadedModule[75]=C:\Windows\SYSTEM32\VERSION.dll LoadedModule[76]=C:\Windows\System32\inetsrv\authsspi.dll LoadedModule[77]=C:\Windows\SYSTEM32\wkscli.dll LoadedModule[78]=C:\Windows\SYSTEM32\netutils.dll LoadedModule[79]=C:\Windows\System32\inetsrv\iiswsock.dll LoadedModule[80]=C:\Windows\SYSTEM32\websocket.dll LoadedModule[81]=C:\Windows\System32\CRYPT32.dll LoadedModule[82]=C:\Windows\System32\MSASN1.dll LoadedModule[83]=C:\Windows\SYSTEM32\DPAPI.DLL LoadedModule[84]=C:\Windows\system32\inetsrv\gzip.dll LoadedModule[85]=C:\Program Files\IIS\Asp.Net Core Module\V2\12.2.18316.0\aspnetcorev2_outofprocess.dll LoadedModule[86]=C:\Windows\SYSTEM32\webio.dll LoadedModule[87]=C:\Windows\SYSTEM32\WINNSI.DLL State[0].Key=Transport.DoneStage1 State[0].Value=1 FriendlyEventName=Stopped working ConsentKey=BEX64 AppName=IIS Worker Process AppPath=c:\windows\system32\inetsrv\w3wp.exe NsPartner=windows NsGroup=windows8 ApplicationIdentity=FCBB9E96A3E01E440E0E4D9368EBD4AC MetadataHash=-1993205964

muratg commented 5 years ago

How often does it crash? Does it work fine without ANCM/IIS (Kestrel only)?

pakrym commented 5 years ago

Do you see any event log messages? The same failure was observed in https://github.com/aspnet/AspNetCore/issues/4413 do you have a bitness mismatch between the app and IIS application pool?

Giorgi commented 5 years ago

@pakrym #4413 used InProcess hosting model but we are using OutOfProcess because we target full .Net Framework and can't use InProcess so I don't think it's the same issue.

Giorgi commented 5 years ago

I configured WER to take full dump on process crash and I have crash dumps of IIS. How can I share it with you?

Giorgi commented 5 years ago

Here is the stacktrace from dump:

ucrtbase!abort+0x4e aspnetcorev2_outofprocess!CreateApplication+0x1be41 aspnetcorev2!RegisterModule+0x3c1f iiscore!NOTIFICATION_MAIN::`scalar deleting destructor'+0x6f iiscore!NOTIFICATION_CONTEXT::DereferenceNotificationContext+0x4d iiscore!W3_CONTEXT_BASE::FinishNotificationLoop+0x52 iiscore!W3_CONTEXT_BASE::OnPostedCompletion+0xc234 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x73 w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33 w3tp!THREAD_MANAGER::ThreadManagerThread+0x49 kernel32!BaseThreadInitThunk+0x14 ntdll!RtlUserThreadStart+0x21

Giorgi commented 5 years ago

@muratg @jkotalik Is there anything else I can do to help in diagnosing the issue ?

muratg commented 5 years ago

@pakrym @jkotalik any more thoughts to help diagnose this?

jkotalik commented 5 years ago

@Giorgi if you could email me at my email listed on my profile, we can figure out a place for you to share the dump.

DaniilSokolyuk commented 5 years ago

@muratg Same problem, after moving to 2.2, crashes every day or more but we are on net core

Host (useful for support):
  Version: 2.2.2
  Commit:  a4fd7b2c84

.NET Core SDKs installed:
  2.0.0 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:

Error

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: ucrtbase.DLL, version: 10.0.10586.1412, time stamp: 0x5a646311
Exception code: 0xc0000409
Fault offset: 0x000000000006993e
Faulting process id: 0x10720
Faulting application start time: 0x01d4d756ef25b782
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\ucrtbase.DLL
Report Id: f07f1aba-4368-11e9-80e2-002590066b2b
Faulting package full name: 
Faulting package-relative application ID: 

Report

Version=1
EventType=BEX64
EventTime=131967189612392956
ReportType=2
Consent=1
ReportIdentifier=f07f1abb-4368-11e9-80e2-002590066b2b
IntegratorReportIdentifier=f07f1aba-4368-11e9-80e2-002590066b2b
NsAppName=w3wp.exe
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=w3wp.exe
Sig[1].Name=Application Version
Sig[1].Value=8.5.9600.16384
Sig[2].Name=Application Timestamp
Sig[2].Value=5215df96
Sig[3].Name=Fault Module Name
Sig[3].Value=ucrtbase.DLL
Sig[4].Name=Fault Module Version
Sig[4].Value=10.0.10586.1412
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=5a646311
Sig[6].Name=Exception Offset
Sig[6].Value=000000000006993e
Sig[7].Name=Exception Code
Sig[7].Value=c0000409
Sig[8].Name=Exception Data
Sig[8].Value=0000000000000007
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.3.9600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1049
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=e013
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=e013440e2dba3ede24b3360c0f76aee5
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=b50c
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=b50c0d407a8bb573aa1795c20c3b88a7
UI[2]=c:\windows\system32\inetsrv\w3wp.exe
LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\system32\KERNEL32.DLL
LoadedModule[3]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[5]=C:\Windows\system32\msvcrt.dll
LoadedModule[6]=C:\Windows\SYSTEM32\combase.dll
LoadedModule[7]=c:\windows\system32\inetsrv\iisutil.dll
LoadedModule[8]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[9]=C:\Windows\system32\RPCRT4.dll
LoadedModule[10]=C:\Windows\system32\WS2_32.dll
LoadedModule[11]=C:\Windows\system32\SspiCli.dll
LoadedModule[12]=C:\Windows\SYSTEM32\pcwum.DLL
LoadedModule[13]=C:\Windows\system32\NSI.dll
LoadedModule[14]=C:\Windows\SYSTEM32\kernel.appcore.dll
LoadedModule[15]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[16]=C:\Windows\SYSTEM32\bcryptPrimitives.dll
LoadedModule[17]=C:\Windows\SYSTEM32\ntmarta.dll
LoadedModule[18]=C:\Windows\SYSTEM32\user32.dll
LoadedModule[19]=C:\Windows\system32\GDI32.dll
LoadedModule[20]=c:\windows\system32\inetsrv\w3wphost.dll
LoadedModule[21]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll
LoadedModule[23]=C:\Windows\SYSTEM32\XmlLite.dll
LoadedModule[24]=C:\Windows\SYSTEM32\ktmw32.dll
LoadedModule[25]=c:\windows\system32\inetsrv\IISRES.DLL
LoadedModule[26]=C:\Windows\SYSTEM32\CRYPTSP.dll
LoadedModule[27]=C:\Windows\system32\rsaenh.dll
LoadedModule[28]=C:\Windows\SYSTEM32\bcrypt.dll
LoadedModule[29]=C:\Windows\SYSTEM32\clbcatq.dll
LoadedModule[30]=C:\Windows\system32\mlang.dll
LoadedModule[31]=C:\Windows\system32\inetsrv\iiscore.dll
LoadedModule[32]=c:\windows\system32\inetsrv\W3TP.dll
LoadedModule[33]=c:\windows\system32\inetsrv\w3dt.dll
LoadedModule[34]=C:\Windows\SYSTEM32\HTTPAPI.dll
LoadedModule[35]=C:\Windows\system32\napinsp.dll
LoadedModule[36]=C:\Windows\system32\NLAapi.dll
LoadedModule[37]=C:\Windows\System32\mswsock.dll
LoadedModule[38]=C:\Windows\SYSTEM32\DNSAPI.dll
LoadedModule[39]=C:\Windows\System32\winrnr.dll
LoadedModule[40]=C:\Windows\System32\rasadhlp.dll
LoadedModule[41]=C:\Windows\SYSTEM32\IPHLPAPI.DLL
LoadedModule[42]=C:\Windows\SYSTEM32\WINNSI.DLL
LoadedModule[43]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[44]=C:\Windows\System32\inetsrv\cachuri.dll
LoadedModule[45]=C:\Windows\System32\inetsrv\cachfile.dll
LoadedModule[46]=C:\Windows\System32\inetsrv\cachtokn.dll
LoadedModule[47]=C:\Windows\System32\inetsrv\cachhttp.dll
LoadedModule[48]=C:\Windows\System32\inetsrv\compdyn.dll
LoadedModule[49]=C:\Windows\System32\inetsrv\compstat.dll
LoadedModule[50]=C:\Windows\System32\inetsrv\defdoc.dll
LoadedModule[51]=C:\Windows\System32\inetsrv\dirlist.dll
LoadedModule[52]=C:\Windows\System32\inetsrv\protsup.dll
LoadedModule[53]=C:\Windows\System32\inetsrv\redirect.dll
LoadedModule[54]=C:\Windows\System32\inetsrv\iis_ssi.dll
LoadedModule[55]=C:\Windows\System32\inetsrv\static.dll
LoadedModule[56]=C:\Windows\System32\inetsrv\authanon.dll
LoadedModule[57]=C:\Windows\System32\inetsrv\authcert.dll
LoadedModule[58]=C:\Windows\System32\inetsrv\urlauthz.dll
LoadedModule[59]=C:\Windows\System32\inetsrv\authbas.dll
LoadedModule[60]=C:\Windows\System32\inetsrv\authsspi.dll
LoadedModule[61]=C:\Windows\SYSTEM32\wkscli.dll
LoadedModule[62]=C:\Windows\SYSTEM32\netutils.dll
LoadedModule[63]=C:\Windows\System32\inetsrv\authmd5.dll
LoadedModule[64]=C:\Windows\System32\inetsrv\authmap.dll
LoadedModule[65]=C:\Windows\System32\inetsrv\iprestr.dll
LoadedModule[66]=C:\Windows\System32\inetsrv\diprestr.dll
LoadedModule[67]=C:\Windows\System32\inetsrv\modrqflt.dll
LoadedModule[68]=C:\Windows\System32\inetsrv\logcust.dll
LoadedModule[69]=C:\Windows\System32\inetsrv\custerr.dll
LoadedModule[70]=C:\Windows\System32\inetsrv\loghttp.dll
LoadedModule[71]=C:\Windows\System32\inetsrv\iisetw.dll
LoadedModule[72]=C:\Windows\System32\inetsrv\iisfreb.dll
LoadedModule[73]=C:\Windows\System32\inetsrv\iisreqs.dll
LoadedModule[74]=C:\Windows\System32\inetsrv\isapi.dll
LoadedModule[75]=C:\Windows\System32\inetsrv\filter.dll
LoadedModule[76]=C:\Windows\System32\inetsrv\iiswsock.dll
LoadedModule[77]=C:\Windows\SYSTEM32\websocket.dll
LoadedModule[78]=C:\Windows\System32\inetsrv\warmup.dll
LoadedModule[79]=C:\Windows\System32\inetsrv\validcfg.dll
LoadedModule[80]=C:\Windows\system32\inetsrv\rewrite.dll
LoadedModule[81]=C:\Windows\system32\inetsrv\aspnetcore.dll
LoadedModule[82]=C:\Windows\SYSTEM32\WINHTTP.dll
LoadedModule[83]=C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll
LoadedModule[84]=C:\Windows\SYSTEM32\api-ms-win-crt-runtime-l1-1-0.dll
LoadedModule[85]=C:\Windows\SYSTEM32\api-ms-win-crt-stdio-l1-1-0.dll
LoadedModule[86]=C:\Windows\SYSTEM32\api-ms-win-crt-heap-l1-1-0.dll
LoadedModule[87]=C:\Windows\SYSTEM32\api-ms-win-crt-string-l1-1-0.dll
LoadedModule[88]=C:\Windows\SYSTEM32\api-ms-win-crt-convert-l1-1-0.dll
LoadedModule[89]=C:\Windows\SYSTEM32\api-ms-win-crt-math-l1-1-0.dll
LoadedModule[90]=C:\Windows\SYSTEM32\api-ms-win-crt-locale-l1-1-0.dll
LoadedModule[91]=C:\Windows\system32\SHELL32.dll
LoadedModule[92]=C:\Windows\SYSTEM32\VERSION.dll
LoadedModule[93]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[94]=C:\Windows\SYSTEM32\ucrtbase.DLL
LoadedModule[95]=C:\Windows\system32\CRYPT32.dll
LoadedModule[96]=C:\Windows\system32\MSASN1.dll
LoadedModule[97]=C:\Windows\SYSTEM32\DPAPI.DLL
LoadedModule[98]=C:\Windows\system32\inetsrv\gzip.dll
LoadedModule[99]=C:\Program Files\IIS\Asp.Net Core Module\V2\12.2.19025.0\aspnetcorev2_outofprocess.dll
LoadedModule[100]=C:\Windows\SYSTEM32\api-ms-win-crt-filesystem-l1-1-0.dll
LoadedModule[101]=C:\Windows\SYSTEM32\api-ms-win-crt-utility-l1-1-0.dll
LoadedModule[102]=C:\Windows\SYSTEM32\webio.dll
FriendlyEventName=Stopped working
ConsentKey=BEX64
AppName=IIS Worker Process
AppPath=c:\windows\system32\inetsrv\w3wp.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=E6858B26E53E85603494FB4B0CF9CA93

web.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <location path="." inheritInChildApplications="false">
        <system.webServer>
            <handlers>
                <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
            </handlers>
            <aspNetCore processPath="dotnet"
                        arguments=".\ApplicationName.dll"
                        stdoutLogEnabled="false"
                        stdoutLogFile=".\logs\stdout"
                        hostingModel="OutOfProcess">
                <environmentVariables>
                    <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Production" />
                </environmentVariables>
            </aspNetCore>
        </system.webServer>
    </location>
</configuration>
DaniilSokolyuk commented 5 years ago

@jkotalik I sent you a dump

DaniilSokolyuk commented 5 years ago

@jkotalik @muratg ping we rolled back to aspnetcoremodulev1, everything works without crashes

jkotalik commented 5 years ago

Can you all please install the 2.2.4 hosting bundle. We believe these issues should be resolved.

SuzanneBeauchamp commented 5 years ago

I installed the 2.2.4 hosting bundle. The issue is not resolved.

analogrelay commented 5 years ago

@SuzanneBeauchamp are you seeing the specific stack trace listed above. This one:

ucrtbase!abort+0x4e
aspnetcorev2_outofprocess!CreateApplication+0x1be41
aspnetcorev2!RegisterModule+0x3c1f
iiscore!NOTIFICATION_MAIN::`scalar deleting destructor'+0x6f
iiscore!NOTIFICATION_CONTEXT::DereferenceNotificationContext+0x4d
iiscore!W3_CONTEXT_BASE::FinishNotificationLoop+0x52
iiscore!W3_CONTEXT_BASE::OnPostedCompletion+0xc234
w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x73
w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x33
w3tp!THREAD_MANAGER::ThreadManagerThread+0x49
kernel32!BaseThreadInitThunk+0x14
ntdll!RtlUserThreadStart+0x21

If so, we can investigate further on this thread. If not, can you create a new issue and include any errors from the Windows Event Log in your report, since it would likely be a different problem.

analogrelay commented 5 years ago

Closing this as we haven't heard from you and generally close issues with no response after ~7 days. Please feel free to comment if you're able to get the information we're looking for and we can reopen the issue to investigate further!

samumeur commented 5 years ago

Hello,

I have the same issue with the ucrtbase.dll while I was deploying my ASPNET Core application to an IIS Server. The application works well from Visual Studio with local IIS, but impossible to deploy it to a server.

Could you help me with that? Do you want more information ? edit : and could you reopen the case please?

SuzanneBeauchamp commented 5 years ago

I upgraded to the 2.2.4 hosting bundle and my problems went away.

From: samumeur [mailto:notifications@github.com] Sent: Friday, June 07, 2019 4:43 AM To: aspnet/AspNetCore AspNetCore@noreply.github.com Cc: Suzanne Beauchamp Suzanne.Beauchamp@georgesinc.com; Mention mention@noreply.github.com Subject: Re: [aspnet/AspNetCore] w3wp.exe crashes with Exception code 0xc0000409 (#6834)

Hello,

I have the same issue with the ucrtbase.dll while I was deploying my ASPNET Core application to an IIS Server. The application works well from Visual Studio with local IIS, but impossible to deploy it to a server.

Could you help me with that? Do you want more information ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_aspnet_AspNetCore_issues_6834-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAKXEOIEXPHAA64PMEIQK4GTPZIUTLA5CNFSM4GQ56AZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXFLLCQ-23issuecomment-2D499824010&d=DwMCaQ&c=4_4TlqFawKYJx72D8V8a3DLVDov-vl3oQSB4o-5WJUI&r=d48_wfmhb_gjFgc2IFQqmFNxwgqJ0irKYrGfNp1X4zY&m=Sq4JAksojZXHB_e5Pb1QVehtOHd4uzBZ6gr1R5jxsz8&s=epMAQnXbG-mZFB_SMZRNEXR7ClRTjUIv-NyHPPS1SCU&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AKXEOIEE2XWYDLASY77FB7TPZIUTLANCNFSM4GQ56AZA&d=DwMCaQ&c=4_4TlqFawKYJx72D8V8a3DLVDov-vl3oQSB4o-5WJUI&r=d48_wfmhb_gjFgc2IFQqmFNxwgqJ0irKYrGfNp1X4zY&m=Sq4JAksojZXHB_e5Pb1QVehtOHd4uzBZ6gr1R5jxsz8&s=s-8ICac7ax_VjB-3V09PaVhGn9Nf71DwXNl3e_-xnQA&e=.

bjornsublime commented 5 years ago

Same issue over here as well, I'm running 2.2.5 on IIS / Windows Server 2016. w3wp crashes a few times a day, refering to ucrtbase.dll:

Unhandled exception at 0x00007FFBA37235DD (ucrtbase.dll) in w3wp.exe: 0xC0000005: Access violation reading location 0x0000000000000007.

Dump:

Version=1 EventType=BEX64 EventTime=132053377748878050 ReportType=2 Consent=1 UploadTime=132053377997067704 ReportIdentifier=2dc227dc-91cc-11e9-b80e-005056b35462 IntegratorReportIdentifier=3bc13842-0566-4ee6-bed9-bc20f210e676 NsAppName=w3wp.exe AppSessionGuid=00001698-0000-0016-4f04-d453d425d501 TargetAppId=W:0000f519feec486de87ed73cb92d3cac802400000000!000054443c275eddd9bbe75c6d47cf9c24076b5d785c!w3wp.exe TargetAppVer=2016//07//16:02:27:22!15ee0!w3wp.exe BootId=4294967295 Response.BucketId=14dcba96117898c96cf93f6cddc9dcf5 Response.BucketTable=5 Response.LegacyBucketId=2087769639072947445 Response.type=4 Sig[0].Name=Application Name Sig[0].Value=w3wp.exe Sig[1].Name=Application Version Sig[1].Value=10.0.14393.0 Sig[2].Name=Application Timestamp Sig[2].Value=57899b8a Sig[3].Name=Fault Module Name Sig[3].Value=ucrtbase.dll Sig[4].Name=Fault Module Version Sig[4].Value=10.0.14393.2879 Sig[5].Name=Fault Module Timestamp Sig[5].Value=5c89e9e2 Sig[6].Name=Exception Offset Sig[6].Value=000000000006d4c8 Sig[7].Name=Exception Code Sig[7].Value=c0000409 Sig[8].Name=Exception Data Sig[8].Value=0000000000000005 DynamicSig[1].Name=OS Version DynamicSig[1].Value=10.0.14393.2.0.0.272.7 DynamicSig[2].Name=Locale ID DynamicSig[2].Value=1033 DynamicSig[22].Name=Additional Information 1 DynamicSig[22].Value=8f58 DynamicSig[23].Name=Additional Information 2 DynamicSig[23].Value=8f58bfa0beaa50ec3a1f47c3c038e3ff DynamicSig[24].Name=Additional Information 3 DynamicSig[24].Value=e787 DynamicSig[25].Name=Additional Information 4 DynamicSig[25].Value=e787dd0608e5539f1edd5e9a0d7c91fa UI[2]=c:\windows\system32\inetsrv\w3wp.exe UI[5]=Check online for a solution (recommended) UI[6]=Check for a solution later (recommended) UI[7]=Close UI[8]=IIS Worker Process stopped working and was closed UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available. UI[10]=&Close LoadedModule[0]=c:\windows\system32\inetsrv\w3wp.exe LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll LoadedModule[4]=C:\Windows\System32\msvcrt.dll LoadedModule[5]=C:\Windows\System32\combase.dll LoadedModule[6]=C:\Windows\System32\ucrtbase.dll LoadedModule[7]=C:\Windows\System32\RPCRT4.dll LoadedModule[8]=C:\Windows\System32\bcryptPrimitives.dll LoadedModule[9]=c:\windows\system32\inetsrv\iisutil.dll LoadedModule[10]=C:\Windows\System32\advapi32.dll LoadedModule[11]=C:\Windows\System32\sechost.dll LoadedModule[12]=C:\Windows\System32\WS2_32.dll LoadedModule[13]=C:\Windows\System32\kernel.appcore.dll LoadedModule[14]=C:\Windows\SYSTEM32\ntmarta.dll LoadedModule[15]=C:\Windows\System32\user32.dll LoadedModule[16]=C:\Windows\System32\win32u.dll LoadedModule[17]=C:\Windows\System32\GDI32.dll LoadedModule[18]=C:\Windows\System32\gdi32full.dll LoadedModule[19]=c:\windows\system32\inetsrv\w3wphost.dll LoadedModule[20]=C:\Windows\System32\OLEAUT32.dll LoadedModule[21]=C:\Windows\System32\msvcp_win.dll LoadedModule[22]=c:\windows\system32\inetsrv\nativerd.dll LoadedModule[23]=C:\Windows\SYSTEM32\ktmw32.dll LoadedModule[24]=C:\Windows\SYSTEM32\ncrypt.dll LoadedModule[25]=C:\Windows\SYSTEM32\XmlLite.dll LoadedModule[26]=C:\Windows\SYSTEM32\bcrypt.dll LoadedModule[27]=C:\Windows\SYSTEM32\NTASN1.dll LoadedModule[28]=c:\windows\system32\inetsrv\IISRES.DLL LoadedModule[29]=C:\Windows\SYSTEM32\CRYPTSP.dll LoadedModule[30]=C:\Windows\system32\rsaenh.dll LoadedModule[31]=C:\Windows\SYSTEM32\CRYPTBASE.dll LoadedModule[32]=C:\Windows\System32\clbcatq.dll LoadedModule[33]=C:\Windows\system32\mlang.dll LoadedModule[34]=C:\Windows\system32\inetsrv\iiscore.dll LoadedModule[35]=C:\Windows\SYSTEM32\SspiCli.dll LoadedModule[36]=c:\windows\system32\inetsrv\W3TP.dll LoadedModule[37]=c:\windows\system32\inetsrv\w3dt.dll LoadedModule[38]=C:\Windows\SYSTEM32\HTTPAPI.dll LoadedModule[39]=C:\Windows\system32\napinsp.dll LoadedModule[40]=C:\Windows\System32\mswsock.dll LoadedModule[41]=C:\Windows\SYSTEM32\DNSAPI.dll LoadedModule[42]=C:\Windows\System32\NSI.dll LoadedModule[43]=C:\Windows\SYSTEM32\IPHLPAPI.DLL LoadedModule[44]=C:\Windows\System32\winrnr.dll LoadedModule[45]=C:\Windows\system32\NLAapi.dll LoadedModule[46]=C:\Windows\System32\rasadhlp.dll LoadedModule[47]=C:\Windows\System32\fwpuclnt.dll LoadedModule[48]=C:\Windows\System32\inetsrv\cachuri.dll LoadedModule[49]=C:\Windows\System32\inetsrv\cachfile.dll LoadedModule[50]=C:\Windows\System32\inetsrv\cachtokn.dll LoadedModule[51]=C:\Windows\System32\inetsrv\cachhttp.dll LoadedModule[52]=C:\Windows\System32\inetsrv\compdyn.dll LoadedModule[53]=C:\Windows\System32\inetsrv\compstat.dll LoadedModule[54]=C:\Windows\System32\inetsrv\defdoc.dll LoadedModule[55]=C:\Windows\System32\inetsrv\protsup.dll LoadedModule[56]=C:\Windows\System32\inetsrv\redirect.dll LoadedModule[57]=C:\Windows\System32\inetsrv\static.dll LoadedModule[58]=C:\Windows\System32\inetsrv\authanon.dll LoadedModule[59]=C:\Windows\System32\inetsrv\authbas.dll LoadedModule[60]=C:\Windows\System32\inetsrv\authsspi.dll LoadedModule[61]=C:\Windows\SYSTEM32\wkscli.dll LoadedModule[62]=C:\Windows\SYSTEM32\netutils.dll LoadedModule[63]=C:\Windows\System32\inetsrv\iprestr.dll LoadedModule[64]=C:\Windows\System32\inetsrv\diprestr.dll LoadedModule[65]=C:\Windows\System32\inetsrv\modrqflt.dll LoadedModule[66]=C:\Windows\System32\inetsrv\isapi.dll LoadedModule[67]=C:\Windows\System32\inetsrv\filter.dll LoadedModule[68]=C:\Windows\System32\inetsrv\iiswsock.dll LoadedModule[69]=C:\Windows\SYSTEM32\websocket.dll LoadedModule[70]=C:\Windows\System32\inetsrv\validcfg.dll LoadedModule[71]=C:\Windows\system32\inetsrv\rewrite.dll LoadedModule[72]=C:\Program Files\IIS\Application Request Routing\requestRouter.dll LoadedModule[73]=C:\Windows\System32\ole32.dll LoadedModule[74]=C:\Windows\System32\CRYPT32.dll LoadedModule[75]=C:\Windows\System32\MSASN1.dll LoadedModule[76]=C:\Windows\SYSTEM32\WINHTTP.dll LoadedModule[77]=C:\Windows\SYSTEM32\pdh.dll LoadedModule[78]=C:\Program Files\IIS\Application Request Routing\gzip.dll LoadedModule[79]=C:\Windows\SYSTEM32\ondemandconnroutehelper.dll LoadedModule[80]=C:\Windows\system32\inetsrv\aspnetcore.dll LoadedModule[81]=C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll LoadedModule[82]=C:\Windows\System32\SHELL32.dll LoadedModule[83]=C:\Windows\System32\cfgmgr32.dll LoadedModule[84]=C:\Windows\System32\windows.storage.dll LoadedModule[85]=C:\Windows\System32\powrprof.dll LoadedModule[86]=C:\Windows\System32\shlwapi.dll LoadedModule[87]=C:\Windows\System32\shcore.dll LoadedModule[88]=C:\Windows\System32\profapi.dll LoadedModule[89]=C:\Windows\SYSTEM32\VERSION.dll LoadedModule[90]=C:\Windows\System32\DPAPI.DLL LoadedModule[91]=C:\Windows\system32\inetsrv\gzip.dll LoadedModule[92]=C:\Program Files\dotnet\host\fxr\2.2.5\hostfxr.dll LoadedModule[93]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\hostpolicy.dll LoadedModule[94]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\aspnetcorev2_inprocess.dll LoadedModule[95]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\coreclr.dll LoadedModule[96]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Private.CoreLib.dll LoadedModule[97]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\clrjit.dll LoadedModule[98]=C:\Octopus\Applications\Development\PRI-identityserver\1.0.0.29-master-8827dec1\IdSrv.dll LoadedModule[99]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.dll LoadedModule[100]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Hosting.Abstractions.dll LoadedModule[101]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\netstandard.dll LoadedModule[102]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Hosting.dll LoadedModule[103]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.dll LoadedModule[104]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Logging.dll LoadedModule[105]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.Abstractions.dll LoadedModule[106]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.DependencyInjection.Abstractions.dll LoadedModule[107]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Hosting.Abstractions.dll LoadedModule[108]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.FileProviders.Abstractions.dll LoadedModule[109]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Collections.dll LoadedModule[110]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.dll LoadedModule[111]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.EnvironmentVariables.dll LoadedModule[112]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Primitives.dll LoadedModule[113]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.Tasks.dll LoadedModule[114]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.Extensions.dll LoadedModule[115]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Linq.dll LoadedModule[116]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.IO.FileSystem.dll LoadedModule[117]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.CommandLine.dll LoadedModule[118]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Server.Kestrel.Core.dll LoadedModule[119]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.dll LoadedModule[120]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Server.Kestrel.dll LoadedModule[121]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Server.IIS.dll LoadedModule[122]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.InteropServices.RuntimeInformation.dll LoadedModule[123]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Server.IISIntegration.dll LoadedModule[124]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.DependencyInjection.dll LoadedModule[125]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.ComponentModel.dll LoadedModule[126]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Logging.Abstractions.dll LoadedModule[127]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Http.Features.dll LoadedModule[128]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Diagnostics.DiagnosticSource.dll LoadedModule[129]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Http.Abstractions.dll LoadedModule[130]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Http.dll LoadedModule[131]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.ObjectPool.dll LoadedModule[132]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.Core.dll LoadedModule[133]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Razor.Runtime.dll LoadedModule[134]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Resources.ResourceManager.dll LoadedModule[135]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.FileProviders.Physical.dll LoadedModule[136]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.FileExtensions.dll LoadedModule[137]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.Json.dll LoadedModule[138]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.dll LoadedModule[139]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.IO.FileSystem.Watcher.dll LoadedModule[140]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.ComponentModel.Primitives.dll LoadedModule[141]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Collections.Concurrent.dll LoadedModule[142]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.FileSystemGlobbing.dll LoadedModule[143]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.Timer.dll LoadedModule[144]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Diagnostics.Tracing.dll LoadedModule[145]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.Overlapped.dll LoadedModule[146]=C:\Octopus\Applications\Development\PRI-identityserver\1.0.0.29-master-8827dec1\Newtonsoft.Json.dll LoadedModule[147]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Linq.Expressions.dll LoadedModule[148]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.ComponentModel.TypeConverter.dll LoadedModule[149]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.ObjectModel.dll LoadedModule[150]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.Numerics.dll LoadedModule[151]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Private.Uri.dll LoadedModule[152]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.InteropServices.dll LoadedModule[153]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Options.dll LoadedModule[154]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll LoadedModule[155]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Hosting.Server.Abstractions.dll LoadedModule[156]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Logging.Configuration.dll LoadedModule[157]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Options.ConfigurationExtensions.dll LoadedModule[158]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Logging.Console.dll LoadedModule[159]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Logging.Debug.dll LoadedModule[160]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Logging.EventSource.dll LoadedModule[161]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.HostFiltering.dll LoadedModule[162]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authentication.Core.dll LoadedModule[163]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authentication.Abstractions.dll LoadedModule[164]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Claims.dll LoadedModule[165]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authentication.dll LoadedModule[166]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.dll LoadedModule[167]=C:\Octopus\Applications\Development\PRI-identityserver\1.0.0.29-master-8827dec1\IdentityServer4.dll LoadedModule[168]=C:\Octopus\Applications\Development\PRI-identityserver\1.0.0.29-master-8827dec1\IdentityServer4.Storage.dll LoadedModule[169]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Cryptography.X509Certificates.dll LoadedModule[170]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authentication.OpenIdConnect.dll LoadedModule[171]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.IdentityModel.Protocols.OpenIdConnect.dll LoadedModule[172]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Cryptography.Primitives.dll LoadedModule[173]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.DependencyModel.dll LoadedModule[174]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Reflection.dll LoadedModule[175]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.AppContext.dll LoadedModule[176]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.IO.FileSystem.Primitives.dll LoadedModule[177]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.IO.dll LoadedModule[178]=C:\Octopus\Applications\Development\PRI-identityserver\1.0.0.29-master-8827dec1\IdSrv.Views.dll LoadedModule[179]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.Razor.dll LoadedModule[180]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Identity.UI.Views.V3.dll LoadedModule[181]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.RazorPages.dll LoadedModule[182]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Identity.UI.Views.V4.dll LoadedModule[183]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Identity.UI.dll LoadedModule[184]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.SpaServices.dll LoadedModule[185]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.SpaServices.Extensions.dll LoadedModule[186]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Routing.dll LoadedModule[187]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Routing.Abstractions.dll LoadedModule[188]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.Abstractions.dll LoadedModule[189]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Buffers.dll LoadedModule[190]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.ApiExplorer.dll LoadedModule[191]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authorization.dll LoadedModule[192]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authorization.Policy.dll LoadedModule[193]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.TagHelpers.dll LoadedModule[194]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Razor.dll LoadedModule[195]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.ViewFeatures.dll LoadedModule[196]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.DataAnnotations.dll LoadedModule[197]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.Formatters.Json.dll LoadedModule[198]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.DataProtection.dll LoadedModule[199]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.DataProtection.Abstractions.dll LoadedModule[200]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Cryptography.Internal.dll LoadedModule[201]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Antiforgery.dll LoadedModule[202]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.WebEncoders.dll LoadedModule[203]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\System.Text.Encodings.Web.dll LoadedModule[204]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.CodeAnalysis.Razor.dll LoadedModule[205]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Razor.Language.dll LoadedModule[206]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.CodeAnalysis.dll LoadedModule[207]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.Razor.Extensions.dll LoadedModule[208]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Caching.Abstractions.dll LoadedModule[209]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Caching.Memory.dll LoadedModule[210]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Mvc.Cors.dll LoadedModule[211]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Cors.dll LoadedModule[212]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authentication.Cookies.dll LoadedModule[213]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Net.Http.dll LoadedModule[214]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.IdentityModel.Tokens.dll LoadedModule[215]=C:\Octopus\Applications\Development\PRI-identityserver\1.0.0.29-master-8827dec1\IdentityModel.dll LoadedModule[216]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Collections.NonGeneric.dll LoadedModule[217]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Cryptography.Encoding.dll LoadedModule[218]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Console.dll LoadedModule[219]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.Thread.dll LoadedModule[220]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Configuration.Binder.dll LoadedModule[221]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Collections.Specialized.dll LoadedModule[222]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Drawing.Primitives.dll LoadedModule[223]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Memory.dll LoadedModule[224]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\Microsoft.Win32.Registry.dll LoadedModule[225]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Principal.Windows.dll LoadedModule[226]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Principal.dll LoadedModule[227]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Xml.XDocument.dll LoadedModule[228]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Private.Xml.Linq.dll LoadedModule[229]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Private.Xml.dll LoadedModule[230]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Cryptography.Algorithms.dll LoadedModule[231]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Text.Encoding.Extensions.dll LoadedModule[232]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Text.RegularExpressions.dll LoadedModule[233]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Reflection.Emit.ILGeneration.dll LoadedModule[234]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Reflection.Emit.Lightweight.dll LoadedModule[235]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Reflection.Primitives.dll LoadedModule[236]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.StaticFiles.dll LoadedModule[237]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Diagnostics.dll LoadedModule[238]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Extensions.Localization.Abstractions.dll LoadedModule[239]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.JsonPatch.dll LoadedModule[240]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.Net.Http.Headers.dll LoadedModule[241]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Collections.Immutable.dll LoadedModule[242]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.Serialization.Formatters.dll LoadedModule[243]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Diagnostics.TraceSource.dll LoadedModule[244]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Net.Primitives.dll LoadedModule[245]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.ThreadPool.dll LoadedModule[246]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Threading.Tasks.Extensions.dll LoadedModule[247]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\System.IO.Pipelines.dll LoadedModule[248]=C:\Windows\System32\Normaliz.dll LoadedModule[249]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Http.Extensions.dll LoadedModule[250]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Authentication.OAuth.dll LoadedModule[251]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\System.IdentityModel.Tokens.Jwt.dll LoadedModule[252]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Xml.ReaderWriter.dll LoadedModule[253]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.IdentityModel.Logging.dll LoadedModule[254]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Net.Security.dll LoadedModule[255]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.IdentityModel.Protocols.dll LoadedModule[256]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.WebUtilities.dll LoadedModule[257]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Runtime.Serialization.Primitives.dll LoadedModule[258]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Data.Common.dll LoadedModule[259]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Html.Abstractions.dll LoadedModule[260]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.CodeAnalysis.CSharp.dll LoadedModule[261]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.AspNetCore.Diagnostics.Abstractions.dll LoadedModule[262]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Diagnostics.Debug.dll LoadedModule[263]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.ComponentModel.Annotations.dll LoadedModule[264]=C:\Windows\system32\msv1_0.DLL LoadedModule[265]=C:\Windows\SYSTEM32\NtlmShared.dll LoadedModule[266]=C:\Windows\SYSTEM32\cryptdll.dll LoadedModule[267]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.5\Microsoft.IdentityModel.JsonWebTokens.dll LoadedModule[268]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Cryptography.Cng.dll LoadedModule[269]=C:\Windows\system32\ncryptprov.dll LoadedModule[270]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Security.Cryptography.Csp.dll LoadedModule[271]=C:\Windows\SYSTEM32\dhcpcsvc6.DLL LoadedModule[272]=C:\Windows\SYSTEM32\dhcpcsvc.DLL LoadedModule[273]=C:\Windows\SYSTEM32\WINNSI.DLL LoadedModule[274]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Net.Sockets.dll LoadedModule[275]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\Microsoft.Win32.Primitives.dll LoadedModule[276]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.5\System.Net.NameResolution.dll LoadedModule[277]=C:\Windows\system32\schannel.DLL LoadedModule[278]=C:\Windows\SYSTEM32\mskeyprotect.dll LoadedModule[279]=C:\Windows\system32\ncryptsslp.dll LoadedModule[280]=C:\Windows\SYSTEM32\cryptnet.dll State[0].Key=Transport.DoneStage1 State[0].Value=1 FriendlyEventName=Stopped working ConsentKey=BEX64 AppName=IIS Worker Process AppPath=c:\windows\system32\inetsrv\w3wp.exe NsPartner=windows NsGroup=windows8 ApplicationIdentity=FCBB9E96A3E01E440E0E4D9368EBD4AC MetadataHash=-1647639642

samumeur commented 5 years ago

Hello,

Actually it was a reference to an old (and non longer existing) packet on another project, so when I removed it, the website worked !

Thanks for your help

Mich-b commented 4 years ago

Same issue over here as well, I'm running 2.2.5 on IIS / Windows Server 2016. w3wp crashes a few times a day, refering to ucrtbase.dll:

Same issue here, with 2.2.5 on IIS / Windows Server 2019.

Any chance this can be reopened? @bjornsublime , did you find a solution?

bjornsublime commented 4 years ago

Same issue over here as well, I'm running 2.2.5 on IIS / Windows Server 2016. w3wp crashes a few times a day, refering to ucrtbase.dll:

Same issue here, with 2.2.5 on IIS / Windows Server 2019.

Any chance this can be reopened? @bjornsublime , did you find a solution?

Not really, but I've narrowed it down a bit. The exception is only thrown during recycling of the app pool I my case. That should not happen either way though in my opinion.

SuzanneBeauchamp commented 4 years ago

I agree that the exception happens upon recycling of app pool.

From: bjornsublime [mailto:notifications@github.com] Sent: Tuesday, November 26, 2019 10:54 AM To: aspnet/AspNetCore AspNetCore@noreply.github.com Cc: Suzanne Beauchamp Suzanne.Beauchamp@georgesinc.com; Mention mention@noreply.github.com Subject: Re: [aspnet/AspNetCore] w3wp.exe crashes with Exception code 0xc0000409 (#6834)

Same issue over here as well, I'm running 2.2.5 on IIS / Windows Server 2016. w3wp crashes a few times a day, refering to ucrtbase.dll:

Same issue here, with 2.2.5 on IIS / Windows Server 2019.

Any chance this can be reopened? @bjornsublimehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bjornsublime&d=DwMCaQ&c=4_4TlqFawKYJx72D8V8a3DLVDov-vl3oQSB4o-5WJUI&r=d48_wfmhb_gjFgc2IFQqmFNxwgqJ0irKYrGfNp1X4zY&m=Gk0HZ-TMBQ9gwlSSvUz1FynWiBrUQcaZACMazkTkPk4&s=NA7WgLekcFs4_KDFJFBCaEB2BnVrRDjw-VCYneAzQTw&e= , did you find a solution?

Not really, but I've narrowed it down a bit. The exception is only thrown during recycling of the app pool I my case. That should not happen either way though in my opinion.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_aspnet_AspNetCore_issues_6834-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAKXEOIBFMYKD7X5MXFXEKCTQVVIDZA5CNFSM4GQ56AZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFGWTFY-23issuecomment-2D558721431&d=DwMCaQ&c=4_4TlqFawKYJx72D8V8a3DLVDov-vl3oQSB4o-5WJUI&r=d48_wfmhb_gjFgc2IFQqmFNxwgqJ0irKYrGfNp1X4zY&m=Gk0HZ-TMBQ9gwlSSvUz1FynWiBrUQcaZACMazkTkPk4&s=UTEyqDjR5UmNGvvb4jAG1piLGK2vGEF1UhlOljX0HvU&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AKXEOIECR7WFEGKUCHPOOLTQVVIDZANCNFSM4GQ56AZA&d=DwMCaQ&c=4_4TlqFawKYJx72D8V8a3DLVDov-vl3oQSB4o-5WJUI&r=d48_wfmhb_gjFgc2IFQqmFNxwgqJ0irKYrGfNp1X4zY&m=Gk0HZ-TMBQ9gwlSSvUz1FynWiBrUQcaZACMazkTkPk4&s=kenbJqIwQmsuXhwLYslyccQj_2QqhVUg2MwjWGWa41c&e=.