search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.55k stars 10.05k forks source link

w3wp.exe reports 1000s of PATH NOT FOUND and Windows Process Activation Service shuts down #8036

Closed shapeh closed 4 years ago

shapeh commented 5 years ago

Describe the bug

I have been running Process Monitor (procmon) on w3wp.exe and I am getting 1000's of PATH NOT FOUND.

w3wp.exe is calling CreateFile several 1000 times in short time (1-2 minutes) on several pages / Controllers in my ASP.NET Core 2.2.2 MVC project. Eventually the app pool will stop with the message:

A process serving application pool 'www.example.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4952'. The data field contains the error number. Note. I am not sure these two things are related but it looks very suspicious.

In my case, root is c:\inetpub\wwwroot\www.example.com

Procmon returns:

"Time of Day","Process Name","PID","Operation","Path","Result","Detail"
"5:12:25.8208570 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo","NAME NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8210456 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8213898 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8215363 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8216481 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8217366 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8218331 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8219342 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8220240 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8221208 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8222883 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m\my-title-1234","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8224565 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m\my-title-1234\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"

Since these paths do not exists (and should not), procmon reports PATH NOT FOUND and NAME NOT FOUND.

To Reproduce

Steps to reproduce the behavior:

  1. Using this version of ASP.NET Core 2.2.2.
  2. No special code -> standard controllers with Dapper, SQL Server 2016, Windows Server 2016, In-Process. I am a bit a loss on where to track down this bug.

Additional context

web.config

 <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" requireAccess="Script" />
    </handlers>
    <aspNetCore processPath="dotnet" arguments=".\example.dll exec &quot;C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll&quot;" stdoutLogEnabled="true" stdoutLogFile=".\logs\log" hostingModel="InProcess">
      <handlerSettings>
        <handlerSetting name="debugLevel" value="file" />
        <handlerSetting name="debugFile" value=".\logs\ancm.log" />
      </handlerSettings>
      <environmentVariables>
        <environmentVariable name="ASPNETCORE_HTTPS_PORT" value="443" />
        <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="PRODUCTION" />
      </environmentVariables>
    </aspNetCore>

ancm.log

[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\sites\www.example.com\'
[aspnetcorev2_inprocess.dll] Starting file watcher thread
[aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 6008.. File Version: 12.2.18346.1. Description: IIS ASP.NET Core Module V2. Commit: 0f9ad16b096ca2535d77efd2ad27645449421b44.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\example.dll exec "C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll"' path: 'C:\inetpub\sites\www.example.com\'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2.dll] Resolving absolute path to dotnet.exe from 'dotnet.exe'
[aspnetcorev2.dll] Invoking where.exe to find dotnet.exe
[aspnetcorev2.dll] where.exe invocation returned: 'C:\Program Files\dotnet\dotnet.exe
C:\Program Files (x86)\dotnet\dotnet.exe
'
[aspnetcorev2.dll] Current process bitness type detected as isX64=1
[aspnetcorev2.dll] Processing entry 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Binary type 6
[aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll'
[aspnetcorev2.dll] Converted argument '.\example.dll' to 'C:\inetpub\sites\www.example.com\.\example.dll'
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Argument[1] = 'C:\inetpub\sites\www.example.com\.\example.dll'
[aspnetcorev2.dll] Argument[2] = 'exec'
[aspnetcorev2.dll] Argument[3] = 'C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll'
[aspnetcorev2.dll] c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020
[aspnetcorev2.dll] Loading request handler:  'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.2\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.2\aspnetcorev2_inprocess.dll'. Process Id: 6008.. File Version: 12.2.19024.2. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 522705f9a27b99ca4ad261f2e89fe51a77b2338e.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\example.dll exec "C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll"' path: 'C:\inetpub\sites\www.example.com\'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2_inprocess.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll'
[aspnetcorev2_inprocess.dll] Converted argument '.\example.dll' to 'C:\inetpub\sites\www.example.com\.\example.dll'
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] Argument[1] = 'C:\inetpub\sites\www.example.com\.\example.dll'
[aspnetcorev2_inprocess.dll] Argument[2] = 'exec'
[aspnetcorev2_inprocess.dll] Argument[3] = 'C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll'
[aspnetcorev2_inprocess.dll] Initial Dll directory: '', current directory: 'c:\windows\system32\inetsrv'
[aspnetcorev2_inprocess.dll] Setting dll directory to c:\windows\system32\inetsrv
[aspnetcorev2_inprocess.dll] Setting current directory to C:\inetpub\sites\www.example.com\
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'C:\inetpub\sites\www.example.com\' started the coreclr in-process successfully.'

dotnet --info

[aspnetcorev2.dll] Current process bitness type detected as isX64=1 [aspnetcorev2.dll] Processing entry 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2.dll] Binary type 6 [aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll' [aspnetcorev2.dll] Converted argument '.\example.dll' to 'C:\inetpub\sites\www.example.com.\example.dll' [aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll' arguments: [aspnetcorev2.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2.dll] Argument[1] = 'C:\inetpub\sites\www.example.com.\example.dll' [aspnetcorev2.dll] Argument[2] = 'exec' [aspnetcorev2.dll] Argument[3] = 'C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll' [aspnetcorev2.dll] c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020 [aspnetcorev2.dll] Loading request handler: 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.2\aspnetcorev2_inprocess.dll' [aspnetcorev2.dll] Creating handler application [aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.2\aspnetcorev2_inprocess.dll'. Process Id: 5588.. File Version: 12.2.19024.2. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 522705f9a27b99ca4ad261f2e89fe51a77b2338e. [aspnetcorev2_inprocess.dll] Waiting for initialization [aspnetcorev2_inprocess.dll] Starting in-process worker thread [aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\example.dll exec "C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll"' path: 'C:\inetpub\sites\www.example.com\' [aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable [aspnetcorev2_inprocess.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2_inprocess.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll' [aspnetcorev2_inprocess.dll] Converted argument '.\example.dll' to 'C:\inetpub\sites\www.example.com.\example.dll' [aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\2.2.2\hostfxr.dll' arguments: [aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe' [aspnetcorev2_inprocess.dll] Argument[1] = 'C:\inetpub\sites\www.example.com.\example.dll' [aspnetcorev2_inprocess.dll] Argument[2] = 'exec' [aspnetcorev2_inprocess.dll] Argument[3] = 'C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll' [aspnetcorev2_inprocess.dll] Initial Dll directory: '', current directory: 'c:\windows\system32\inetsrv' [aspnetcorev2_inprocess.dll] Setting dll directory to c:\windows\system32\inetsrv [aspnetcorev2_inprocess.dll] Setting current directory to C:\inetpub\sites\www.example.com\ [aspnetcorev2_inprocess.dll] In-process callbacks set [aspnetcorev2_inprocess.dll] Event Log: 'Application 'C:\inetpub\sites\www.example.com\' started the coreclr in-process successfully.' End Event Log Message. [aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\sites\www.example.com\' [aspnetcorev2_inprocess.dll] Starting file watcher thread

.NET Core SDK (reflecting any global.json): Version: 2.2.104 Commit: 73f036d4ac

Runtime Environment: OS Name: Windows OS Version: 10.0.14393 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\2.2.104\

Host (useful for support): Version: 2.2.2 Commit: a4fd7b2c84

.NET Core SDKs installed: 2.1.300 [C:\Program Files\dotnet\sdk] 2.1.301 [C:\Program Files\dotnet\sdk] 2.2.100 [C:\Program Files\dotnet\sdk] 2.2.104 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

shapeh commented 5 years ago

While https://github.com/aspnet/AspNetCore/issues/6118 is closed, the ancm log on that issue also shows

[aspnetcorev2.dll] c:\b\w\e37dd45d8cd1eaf4\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020

Here it is slight different (da744fbcc13abce\src instead of e37dd45d8cd1eaf4\src)

[aspnetcorev2.dll] c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020

According to https://github.com/aspnet/AspNetCore/issues/6118 this was resolved by https://github.com/aspnet/AspNetCore/issues/4206 but the options there are not working for me.

Ref: https://github.com/aspnet/AspNetCore/issues/4431

shapeh commented 5 years ago

Update: Same thing happens with OutOfProcess.

shapeh commented 5 years ago

@muratg You are probably busy but I wonder if you have inputs on how to debug this? e.g. DebugDiag settings or something else.

muratg commented 5 years ago

Nothing comes to my mind, @shirhatti any thoughts why this may be happening?

@shapeh We may need a simple app that reproduces this issue to get to the bottom of it.

shapeh commented 5 years ago

@muratg - okay re the app. I am not sure how I can produce that for you? It seems to be happening only for this particular app and we run 5-6 other asp.net core apps on same server. The other apps also use dapper, same sql server etc.

Could it be some file watching mechanism / compression / caching thing in IIS we have set up wrongfully or something? As far as I am aware, nothing in our code makes w3wp.exe requests these pages.

web.config (outofprocess):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" requireAccess="Script" />
    </handlers>
    <aspNetCore processPath="dotnet" arguments=".\example.dll" stdoutLogEnabled="true" stdoutLogFile=".\logs\log" hostingModel="outofprocess">
      <handlerSettings>
        <handlerSetting name="debugLevel" value="file" />
        <handlerSetting name="debugFile" value=".\logs\ancm.log" />
      </handlerSettings>
      <environmentVariables>
        <environmentVariable name="ASPNETCORE_HTTPS_PORT" value="443" />
        <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="PRODUCTION" />
      </environmentVariables>
    </aspNetCore>
    <rewrite>
           <rewriteMaps name="legacyredirects">
                 <rewriteMap>
        <!-- a long list of keys here for redirect rule -->
                 </rewriteMap>
           </rewriteMaps>
        <!-- a long list of redirect rules here -->
    </rewrite>
    <httpProtocol>
      <customHeaders>
        <remove name="X-Powered-By" />
      </customHeaders>
    </httpProtocol>
    <tracing>
      <traceFailedRequests>
        <add path="*">
          <traceAreas>
            <add provider="ASP" verbosity="Verbose" />
            <add provider="ASPNET" areas="Infrastructure,Module,Page,AppServices" verbosity="Verbose" />
            <add provider="ISAPI Extension" verbosity="Verbose" />
            <add provider="WWW Server" areas="Authentication,Security,Filter,StaticFile,CGI,Compression,Cache,RequestNotifications,Module,FastCGI,WebSocket" verbosity="Verbose" />
          </traceAreas>
          <failureDefinitions timeTaken="00:00:05" statusCodes="500-999" />
        </add>
      </traceFailedRequests>
    </tracing>
    <caching>
      <profiles>
        <add extension=".js" policy="CacheUntilChange" kernelCachePolicy="CacheUntilChange" />
        <add extension=".svg" policy="CacheUntilChange" kernelCachePolicy="CacheUntilChange" />
        <add extension=".gif" policy="CacheUntilChange" kernelCachePolicy="CacheUntilChange" />
        <add extension=".jpg" policy="CacheUntilChange" kernelCachePolicy="CacheUntilChange" />
        <add extension=".png" policy="CacheUntilChange" kernelCachePolicy="CacheUntilChange" />
      </profiles>
    </caching>
  </system.webServer>
</configuration>
shirhatti commented 5 years ago

👀

[aspnetcorev2.dll] c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020

HR 0x80070020 is ERROR_SHARING_VIOLATION: The process cannot access the file because it is being used by another process.

shapeh commented 5 years ago

@shirhatti - is this related to the CreateFile issue or are these unrelated errors?

shapeh commented 5 years ago

Just troubleshooting here - looking at our code we are using RouteDataRequestCultureProvider and IRouteConstraint. Could this be causing the issue?

startup.cs

            services.Configure<RouteOptions>(options =>
            {
                options.LowercaseUrls = true;
                options.AppendTrailingSlash = false;
                options.ConstraintMap.Add("lang", typeof(LanguageRouteConstraint));
            });

LanguageRouteConstraint.cs

    public class LanguageRouteConstraint : IRouteConstraint
    {
        private readonly AppLanguages _languageSettings;

        public LanguageRouteConstraint()
        {
            var builder = new ConfigurationBuilder()
                    .SetBasePath(Directory.GetCurrentDirectory())
                    .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);

            IConfigurationRoot configuration = builder.Build();

            _languageSettings = new AppLanguages();
            configuration.GetSection("AppLanguages").Bind(_languageSettings);
        }

        public bool Match(HttpContext httpContext, IRouter route, string routeKey, RouteValueDictionary values, RouteDirection routeDirection)
        {
            if (!values.ContainsKey("lang"))
            {
                return false;
            }

            var lang = values["lang"].ToString();
            foreach (Language lang_in_app in _languageSettings.Dict.Values)
            {
                if (lang == lang_in_app.Icc)
                {
                    return true;
                }
            }
            return false;
        }
    }

RouteDataRequestCultureProvider.cs 

    public class RouteDataRequestCultureProvider : RequestCultureProvider
    {
        /// <summary>
        /// The key that contains the culture name.
        /// Defaults to "culture".
        /// </summary>
        public string RouteDataStringKey { get; set; } = "culture";

        private readonly AppLanguages _langs;

        public RouteDataRequestCultureProvider(AppLanguages langs)
        {
            _langs = langs;
        }

        /// <inheritdoc />
        public override Task<ProviderCultureResult> DetermineProviderCultureResult(HttpContext httpContext)
        {
            if (httpContext == null)
            {
                throw new ArgumentNullException(nameof(httpContext));
            }

            string lang = null;

            if (!string.IsNullOrEmpty(RouteDataStringKey))
            {
                lang = httpContext.GetRouteValue(RouteDataStringKey)?.ToString();
            }

            // it seems we have a naked request like www.example.com (without any language or folder)
            if (string.IsNullOrEmpty(lang))
            {
                // set default lang to "int"
                lang = "int";             

                // detect which languages the user has in the browser
                string acceptLang = httpContext.Request.Headers["Accept-Language"].ToString();
                if (!string.IsNullOrEmpty(acceptLang))
                {
                    // e.g. 
                    // da,en-US;q=0.9,en;q=0.8
                    // fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5
                    // en-US,en;q=0.5
                    // de

                    // try to get first language, e.g. "da" or "de" or "en-US"
                    var firstLang = acceptLang.Split(',').FirstOrDefault();

                    // choose a default in case acceptLanguages is null / empty, then convert to CultureInfo object
                    var cult = new CultureInfo(string.IsNullOrEmpty(firstLang) ? "en-gb" : firstLang);

                    // get the ICC for this culture -> only visible languages (to avoid selecting 'int' for en-us.), match cult.Name (either e.g. "en" or "en-us" against appsettings.json)
                    var acceptLangMatched = _langs.Dict.Values.FirstOrDefault(x => x.Visible == true && (x.Culture == cult.Name.ToLower() || x.Culture.Split('-')[0].ToString() == cult.Name.ToLower()));

                    if (acceptLangMatched != null)
                    {
                        lang = acceptLangMatched.Icc;
                    }
                }
                httpContext.Response.Redirect("/" + lang);
            }

            // set culture based on path icc, e.g. /dk --> da-dk culture
            // if user enters /<non-existing icc>/ page will produce a "500 error" - maybe this could be 404 instead.
            var culture = _langs.Dict.Values.SingleOrDefault(l => l.Icc == lang);

            lang = "int";
            if (culture != null)
            {
                lang = culture.Culture;
                var providerResultCulture = new ProviderCultureResult(lang);

                return Task.FromResult(providerResultCulture);
            }
            else
            {
                //httpContext.Response.StatusCode = 404;
                httpContext.Response.Redirect("/int/404");
                return Task.FromResult(new ProviderCultureResult("uk"));
            }

        }
    }
shirhatti commented 5 years ago 
<aspNetCore processPath="dotnet"
            arguments=".\example.dll exec &quot;C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll&quot;"
            stdoutLogEnabled="true"
            stdoutLogFile=".\logs\log"
            hostingModel="InProcess">

Curious what you're doing here?

The WebSDK should have just generated something like 

<aspNetCore processPath="dotnet"
            arguments=".\example.dll"
            stdoutLogEnabled="true"
            stdoutLogFile=".\logs\log"
            hostingModel="InProcess">
shapeh commented 5 years ago

That additional stuff was added by VS - we started with 1.5-1.6 project and have upgraded. I have removed that since myself. There is no change in the CreateFile events. They keep happening in both inprocess and outofprocess.

I just looked in procmon.exe - this particular call is happening as well: C:\inetpub\sites\www.example.com\example.eventpipeconfig

So now it looks like this for inprocess:

<aspNetCore processPath="dotnet"
            arguments=".\example.dll"
            stdoutLogEnabled="true"
            stdoutLogFile=".\logs\log"
            hostingModel="inprocess">

and this for outofprocess

<aspNetCore processPath="dotnet"
            arguments=".\example.dll"
            stdoutLogEnabled="true"
            stdoutLogFile=".\logs\log"
            hostingModel="outofprocess">

Some more stuff from procmon on the actual CreateFile event.

FASTIO_NETWORK_QUERY_OPEN:

Description:    IIS Worker Process
Company:    Microsoft Corporation
Name:   w3wp.exe
Version:    10.0.14393.0 (rs1_release.160715-1616)
Path:   c:\windows\system32\inetsrv\w3wp.exe
Command Line:   c:\windows\system32\inetsrv\w3wp.exe -ap "www.example.com" -a \\.\pipe\iisipme11f253b-25a8-408b-b421-ae7bd8f6bd34 -h "C:\inetpub\temp\apppools\www.example.com\www.example.com.config" -w "" -m 0 -t 120 -ta 0
PID:    6640
Parent PID: 1504
Session ID: 0
User:   IIS APPPOOL\www.example.com
Auth ID:    00000000:0001aa04
Architecture:   64-bit
Virtualized:    False
Integrity:  High
Started:    3/15/2019 1:38:55 AM
Ended:  (Running)
Modules:
IISRES.DLL  0x2cc09790000   0x3b000 c:\windows\system32\inetsrv\IISRES.DLL  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:24 AM
w3wp.exe    0x7ff6d4a90000  0xb000  c:\windows\system32\inetsrv\w3wp.exe    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:22 AM
diprestr.dll    0x7ffbf9270000  0xe000  C:\Windows\System32\inetsrv\diprestr.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:00 AM
iprestr.dll 0x7ffbf92b0000  0xc000  C:\Windows\System32\inetsrv\iprestr.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:40 AM
authbas.dll 0x7ffbf92c0000  0xe000  C:\Windows\System32\inetsrv\authbas.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:04 AM
iisbrotli.dll   0x7ffbfa380000  0xb4000 C:\Program Files\IIS\IIS Compression\iisbrotli.dll  Microsoft Corporation   1.0.9.0 5/9/2018 1:29:50 AM
aspnetcorev2_outofprocess.dll   0x7ffbfa630000  0x3d000 C:\Program Files\IIS\Asp.Net Core Module\V2\12.2.18346.0\aspnetcorev2_outofprocess.dll  Microsoft   12.2.18346.1    12/12/2018 1:24:10 AM
iiszlib.dll 0x7ffbfa670000  0x22000 C:\Program Files\IIS\IIS Compression\iiszlib.dll    Microsoft Corporation   1.0.9.0 5/9/2018 1:27:57 AM
aspnetcorev2.dll    0x7ffbfaa30000  0x4c000 C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll    Microsoft   12.2.18346.1    12/12/2018 1:24:53 AM
aspnetcore.dll  0x7ffbfab60000  0x42000 C:\Windows\system32\inetsrv\aspnetcore.dll  Microsoft Corporation   12.2.18346.1    12/12/2018 1:24:44 AM
rewrite.dll 0x7ffbfac40000  0x64000 C:\Windows\system32\inetsrv\rewrite.dll Microsoft Corporation   7.1.1980.0  6/7/2017 12:30:58 AM
validcfg.dll    0x7ffbfacb0000  0xa000  C:\Windows\System32\inetsrv\validcfg.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:43 AM
warmup.dll  0x7ffbfacc0000  0xd000  C:\Windows\System32\inetsrv\warmup.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:59 AM
websocket.dll   0x7ffbfacd0000  0xf000  C:\Windows\SYSTEM32\websocket.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:38 AM
filter.dll  0x7ffbfacf0000  0x15000 C:\Windows\System32\inetsrv\filter.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:59 AM
isapi.dll   0x7ffbfad10000  0x24000 C:\Windows\System32\inetsrv\isapi.dll   Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:12:50 AM
compdyn.dll 0x7ffbfad40000  0xe000  C:\Windows\System32\inetsrv\compdyn.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:36 AM
winrnr.dll  0x7ffbfbeb0000  0xe000  C:\Windows\System32\winrnr.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:05 AM
napinsp.dll 0x7ffbfbec0000  0x16000 C:\Windows\system32\napinsp.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:28 AM
w3dt.dll    0x7ffbfccf0000  0x1f000 c:\windows\system32\inetsrv\w3dt.dll    Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:06:18 AM
iiscore.dll 0x7ffbfd740000  0x54000 C:\Windows\system32\inetsrv\iiscore.dll Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:10:40 AM
modrqflt.dll    0x7ffbfd7c0000  0xf000  C:\Windows\System32\inetsrv\modrqflt.dll    Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:13:10 AM
authanon.dll    0x7ffbfd7d0000  0x10000 C:\Windows\System32\inetsrv\authanon.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:39 AM
static.dll  0x7ffbfdc00000  0xf000  C:\Windows\System32\inetsrv\static.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:09 AM
protsup.dll 0x7ffbfdc10000  0xa000  C:\Windows\System32\inetsrv\protsup.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:17 AM
dirlist.dll 0x7ffbfdc20000  0xa000  C:\Windows\System32\inetsrv\dirlist.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:34 AM
compstat.dll    0x7ffbfdc50000  0x11000 C:\Windows\System32\inetsrv\compstat.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:17 AM
cachhttp.dll    0x7ffbfdc70000  0x11000 C:\Windows\System32\inetsrv\cachhttp.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:54 AM
defdoc.dll  0x7ffbfdc90000  0xa000  C:\Windows\System32\inetsrv\defdoc.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:44 AM
cachtokn.dll    0x7ffbfdca0000  0x9000  C:\Windows\System32\inetsrv\cachtokn.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:40 AM
cachfile.dll    0x7ffbfdcb0000  0xa000  C:\Windows\System32\inetsrv\cachfile.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:17 AM
cachuri.dll 0x7ffbfdcc0000  0x9000  C:\Windows\System32\inetsrv\cachuri.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:15 AM
loghttp.dll 0x7ffbfdcd0000  0xe000  C:\Windows\System32\inetsrv\loghttp.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:12 AM
w3wphost.dll    0x7ffbfdd60000  0x17000 c:\windows\system32\inetsrv\w3wphost.dll    Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:49:16 AM
HTTPAPI.dll 0x7ffc01090000  0xd000  C:\Windows\SYSTEM32\HTTPAPI.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/12/2017 6:20:24 AM
W3TP.dll    0x7ffc01470000  0xc000  c:\windows\system32\inetsrv\W3TP.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:13 AM
mlang.dll   0x7ffc03dc0000  0x3f000 C:\Windows\system32\mlang.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/23/2018 4:37:19 AM
VERSION.dll 0x7ffc049a0000  0xa000  C:\Windows\SYSTEM32\VERSION.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:11 AM
ktmw32.dll  0x7ffc04a50000  0xb000  C:\Windows\SYSTEM32\ktmw32.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:10:41 AM
nativerd.dll    0x7ffc04a60000  0x7c000 c:\windows\system32\inetsrv\nativerd.dll    Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:45:02 AM
iisutil.dll 0x7ffc04d50000  0x4d000 c:\windows\system32\inetsrv\iisutil.dll Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:46:54 AM
rasadhlp.dll    0x7ffc057b0000  0xa000  C:\Windows\System32\rasadhlp.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:58 AM
webio.dll   0x7ffc05a30000  0x90000 C:\Windows\SYSTEM32\webio.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/30/2018 9:33:32 PM
ondemandconnroutehelper.dll 0x7ffc05ae0000  0x15000 C:\Windows\SYSTEM32\ondemandconnroutehelper.dll Microsoft Corporation   10.0.14393.351 (rs1_release_inmarket.161014-1755)   10/15/2016 4:56:41 AM
XmlLite.dll 0x7ffc06bd0000  0x36000 C:\Windows\SYSTEM32\XmlLite.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:02 AM
fwpuclnt.dll    0x7ffc06eb0000  0x6a000 C:\Windows\System32\fwpuclnt.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:12:25 AM
WINNSI.DLL  0x7ffc082e0000  0xb000  C:\Windows\SYSTEM32\WINNSI.DLL  Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:49:47 AM
NLAapi.dll  0x7ffc088f0000  0x18000 C:\Windows\system32\NLAapi.dll  Microsoft Corporation   10.0.14393.2636 (rs1_release_1.181031-1836) 11/1/2018 5:25:13 AM
WINHTTP.dll 0x7ffc0a230000  0xcc000 C:\Windows\SYSTEM32\WINHTTP.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:03:27 AM
ntmarta.dll 0x7ffc0c3b0000  0x32000 C:\Windows\SYSTEM32\ntmarta.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:02 AM
rsaenh.dll  0x7ffc0cba0000  0x33000 C:\Windows\system32\rsaenh.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/23/2018 4:38:30 AM
DPAPI.DLL   0x7ffc0cbe0000  0xa000  C:\Windows\SYSTEM32\DPAPI.DLL   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:47 AM
IPHLPAPI.DLL    0x7ffc0cd20000  0x38000 C:\Windows\SYSTEM32\IPHLPAPI.DLL    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  6/12/2018 1:41:33 AM
DNSAPI.dll  0x7ffc0cd60000  0xa1000 C:\Windows\SYSTEM32\DNSAPI.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  6/26/2018 4:11:22 AM
mswsock.dll 0x7ffc0cfa0000  0x5c000 C:\Windows\System32\mswsock.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:22:57 AM
CRYPTSP.dll 0x7ffc0d140000  0x17000 C:\Windows\SYSTEM32\CRYPTSP.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:12 AM
CRYPTBASE.dll   0x7ffc0d160000  0xb000  C:\Windows\SYSTEM32\CRYPTBASE.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:23 AM
NTASN1.dll  0x7ffc0d1e0000  0x3b000 C:\Windows\SYSTEM32\NTASN1.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:32 AM
ncrypt.dll  0x7ffc0d220000  0x26000 C:\Windows\SYSTEM32\ncrypt.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/23/2018 4:38:40 AM
SspiCli.dll 0x7ffc0d340000  0x2c000 C:\Windows\SYSTEM32\SspiCli.dll Microsoft Corporation   10.0.14393.2580 (rs1_release_inmarket.181009-1745)  10/10/2018 8:48:34 AM
bcrypt.dll  0x7ffc0d610000  0x2b000 C:\Windows\SYSTEM32\bcrypt.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  6/8/2018 6:46:34 AM
profapi.dll 0x7ffc0d6d0000  0x14000 C:\Windows\System32\profapi.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:56 AM
powrprof.dll    0x7ffc0d6f0000  0x4c000 C:\Windows\System32\powrprof.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:21:19 AM
kernel.appcore.dll  0x7ffc0d740000  0xf000  C:\Windows\System32\kernel.appcore.dll  Microsoft Corporation   10.0.14393.2312 (rs1_release.180607-1919)   6/8/2018 6:48:18 AM
windows.storage.dll 0x7ffc0d760000  0x6d9000    C:\Windows\System32\windows.storage.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:17:36 AM
ucrtbase.dll    0x7ffc0de40000  0xf4000 C:\Windows\System32\ucrtbase.dll    Microsoft Corporation   10.0.14393.2848 (rs1_release.190305-1856)   3/6/2019 7:01:00 AM
win32u.dll  0x7ffc0df40000  0x1e000 C:\Windows\System32\win32u.dll  Microsoft Corporation   10.0.14393.51 (rs1_release_inmarket.160801-1836)    8/2/2016 9:21:20 AM
CRYPT32.dll 0x7ffc0e010000  0x1c9000    C:\Windows\System32\CRYPT32.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  4/3/2018 4:35:01 AM
gdi32full.dll   0x7ffc0e1e0000  0x181000    C:\Windows\System32\gdi32full.dll   Microsoft Corporation   10.0.14393.2791 (rs1_release.190205-1511)   2/6/2019 3:14:08 AM
shcore.dll  0x7ffc0e370000  0xa9000 C:\Windows\System32\shcore.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:04:31 AM
KERNELBASE.dll  0x7ffc0e420000  0x21d000    C:\Windows\System32\KERNELBASE.dll  Microsoft Corporation   10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)  3/6/2019 7:02:22 AM
msvcp_win.dll   0x7ffc0e6a0000  0x9c000 C:\Windows\System32\msvcp_win.dll   Microsoft Corporation   10.0.14393.2848 (rs1_release.190305-1856)   3/6/2019 7:00:46 AM
cfgmgr32.dll    0x7ffc0e740000  0x42000 C:\Windows\System32\cfgmgr32.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:04 AM
bcryptPrimitives.dll    0x7ffc0e790000  0x6a000 C:\Windows\System32\bcryptPrimitives.dll    Microsoft Corporation   10.0.14393.2457 (rs1_release_inmarket.180822-1743)  8/23/2018 4:37:31 AM
shlwapi.dll 0x7ffc0e800000  0x52000 C:\Windows\System32\shlwapi.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:14:37 AM
advapi32.dll    0x7ffc0e860000  0xa2000 C:\Windows\System32\advapi32.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/30/2018 9:34:36 PM
GDI32.dll   0x7ffc0ed70000  0x34000 C:\Windows\System32\GDI32.dll   Microsoft Corporation   10.0.14393.2848 (rs1_release.190305-1856)   3/6/2019 7:07:25 AM
WS2_32.dll  0x7ffc0edb0000  0x6a000 C:\Windows\System32\WS2_32.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  9/15/2016 5:34:44 PM
KERNEL32.DLL    0x7ffc0ee20000  0xac000 C:\Windows\System32\KERNEL32.DLL    Microsoft Corporation   10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)  3/6/2019 7:16:30 AM
SHELL32.dll 0x7ffc0ef40000  0x1505000   C:\Windows\System32\SHELL32.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:12:44 AM
clbcatq.dll 0x7ffc105e0000  0x9f000 C:\Windows\System32\clbcatq.dll Microsoft Corporation   2001.12.10941.16384 (rs1_release.160715-1616)   7/16/2016 3:18:57 AM
NSI.dll 0x7ffc10960000  0x8000  C:\Windows\System32\NSI.dll Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:49:58 AM
RPCRT4.dll  0x7ffc10970000  0x121000    C:\Windows\System32\RPCRT4.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:01:18 AM
OLEAUT32.dll    0x7ffc10ba0000  0xbf000 C:\Windows\System32\OLEAUT32.dll    Microsoft Corporation   10.0.14393.2791 (rs1_release.190205-1511)   2/6/2019 3:11:09 AM
user32.dll  0x7ffc10d20000  0x165000    C:\Windows\System32\user32.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  4/28/2018 6:41:41 AM
sechost.dll 0x7ffc10ea0000  0x59000 C:\Windows\System32\sechost.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/30/2018 9:34:58 PM
combase.dll 0x7ffc10f00000  0x2c5000    C:\Windows\System32\combase.dll Microsoft Corporation   10.0.14393.1198 (rs1_release_sec.170427-1353)   3/6/2019 7:04:41 AM
msvcrt.dll  0x7ffc111d0000  0x9e000 C:\Windows\System32\msvcrt.dll  Microsoft Corporation   7.0.14393.2457 (rs1_release_inmarket.180822-1743)   8/23/2018 4:39:07 AM
ntdll.dll   0x7ffc11270000  0x1d0000    C:\Windows\SYSTEM32\ntdll.dll   Microsoft Corporation   10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)  10/25/2018 4:09:08 AM

FASTIO_NETWORK_QUERY_OPEN: - stack

0   FLTMGR.SYS  FltDecodeParameters + 0x195a    0xfffff807bc1746ca  C:\Windows\System32\drivers\FLTMGR.SYS
1   FLTMGR.SYS  FltDecodeParameters + 0x1e3 0xfffff807bc172f53  C:\Windows\System32\drivers\FLTMGR.SYS
2   FLTMGR.SYS  FltQueryInformationFile + 0x19e 0xfffff807bc1a3b1e  C:\Windows\System32\drivers\FLTMGR.SYS
3   ntoskrnl.exe    SeQueryInformationToken + 0x7a0c    0xfffff8008db1e4fc  C:\Windows\system32\ntoskrnl.exe
4   ntoskrnl.exe    ObWaitForMultipleObjects + 0xf2f    0xfffff8008db3204f  C:\Windows\system32\ntoskrnl.exe
5   ntoskrnl.exe    ObOpenObjectByNameEx + 0x1dd    0xfffff8008db663dd  C:\Windows\system32\ntoskrnl.exe
6   ntoskrnl.exe    RtlQueryInformationAcl + 0x22c  0xfffff8008daff118  C:\Windows\system32\ntoskrnl.exe
7   ntoskrnl.exe    setjmpex + 0x68a3   0xfffff8008d7f9403  C:\Windows\system32\ntoskrnl.exe
8   ntdll.dll   NtQueryFullAttributesFile + 0x14    0x7ffc11318174  C:\Windows\SYSTEM32\ntdll.dll
9   nativerd.dll    DllGetClassObject + 0x11d6b 0x7ffc04a7303b  c:\windows\system32\inetsrv\nativerd.dll
10  nativerd.dll    DllGetClassObject + 0x142b0 0x7ffc04a75580  c:\windows\system32\inetsrv\nativerd.dll
11  iiscore.dll iiscore.dll + 0xa1a5    0x7ffbfd74a1a5  C:\Windows\system32\inetsrv\iiscore.dll
12  iiscore.dll iiscore.dll + 0xa535    0x7ffbfd74a535  C:\Windows\system32\inetsrv\iiscore.dll
13  iiscore.dll iiscore.dll + 0xb144    0x7ffbfd74b144  C:\Windows\system32\inetsrv\iiscore.dll
14  iiscore.dll iiscore.dll + 0x11bbf   0x7ffbfd751bbf  C:\Windows\system32\inetsrv\iiscore.dll
15  w3dt.dll    UlAtqSendEntityBody + 0xc5  0x7ffbfccf4c95  c:\windows\system32\inetsrv\w3dt.dll
16  W3TP.dll    THREAD_POOL::CreateThreadPool + 0x4f3   0x7ffc01471853  c:\windows\system32\inetsrv\W3TP.dll
17  W3TP.dll    THREAD_POOL::CreateThreadPool + 0x443   0x7ffc014717a3  c:\windows\system32\inetsrv\W3TP.dll
18  W3TP.dll    THREAD_POOL::CreateThreadPool + 0x399   0x7ffc014716f9  c:\windows\system32\inetsrv\W3TP.dll
19  KERNEL32.DLL    BaseThreadInitThunk + 0x14  0x7ffc0ee284d4  C:\Windows\System32\KERNEL32.DLL
20  ntdll.dll   RtlUserThreadStart + 0x21   0x7ffc112de851  C:\Windows\SYSTEM32\ntdll.dll

IRP_MJ_CREATE (CreateFile):

Description:    IIS Worker Process
Company:    Microsoft Corporation
Name:   w3wp.exe
Version:    10.0.14393.0 (rs1_release.160715-1616)
Path:   c:\windows\system32\inetsrv\w3wp.exe
Command Line:   c:\windows\system32\inetsrv\w3wp.exe -ap "www.example.com" -a \\.\pipe\iisipme11f253b-25a8-408b-b421-ae7bd8f6bd34 -h "C:\inetpub\temp\apppools\www.example.com\www.example.com.config" -w "" -m 0 -t 120 -ta 0
PID:    6640
Parent PID: 1504
Session ID: 0
User:   IIS APPPOOL\www.example.com
Auth ID:    00000000:0001aa04
Architecture:   64-bit
Virtualized:    False
Integrity:  High
Started:    3/15/2019 1:38:55 AM
Ended:  (Running)
Modules:
IISRES.DLL  0x2cc09790000   0x3b000 c:\windows\system32\inetsrv\IISRES.DLL  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:24 AM
w3wp.exe    0x7ff6d4a90000  0xb000  c:\windows\system32\inetsrv\w3wp.exe    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:22 AM
diprestr.dll    0x7ffbf9270000  0xe000  C:\Windows\System32\inetsrv\diprestr.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:00 AM
iprestr.dll 0x7ffbf92b0000  0xc000  C:\Windows\System32\inetsrv\iprestr.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:40 AM
authbas.dll 0x7ffbf92c0000  0xe000  C:\Windows\System32\inetsrv\authbas.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:04 AM
iisbrotli.dll   0x7ffbfa380000  0xb4000 C:\Program Files\IIS\IIS Compression\iisbrotli.dll  Microsoft Corporation   1.0.9.0 5/9/2018 1:29:50 AM
aspnetcorev2_outofprocess.dll   0x7ffbfa630000  0x3d000 C:\Program Files\IIS\Asp.Net Core Module\V2\12.2.18346.0\aspnetcorev2_outofprocess.dll  Microsoft   12.2.18346.1    12/12/2018 1:24:10 AM
iiszlib.dll 0x7ffbfa670000  0x22000 C:\Program Files\IIS\IIS Compression\iiszlib.dll    Microsoft Corporation   1.0.9.0 5/9/2018 1:27:57 AM
aspnetcorev2.dll    0x7ffbfaa30000  0x4c000 C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll    Microsoft   12.2.18346.1    12/12/2018 1:24:53 AM
aspnetcore.dll  0x7ffbfab60000  0x42000 C:\Windows\system32\inetsrv\aspnetcore.dll  Microsoft Corporation   12.2.18346.1    12/12/2018 1:24:44 AM
rewrite.dll 0x7ffbfac40000  0x64000 C:\Windows\system32\inetsrv\rewrite.dll Microsoft Corporation   7.1.1980.0  6/7/2017 12:30:58 AM
validcfg.dll    0x7ffbfacb0000  0xa000  C:\Windows\System32\inetsrv\validcfg.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:43 AM
warmup.dll  0x7ffbfacc0000  0xd000  C:\Windows\System32\inetsrv\warmup.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:59 AM
websocket.dll   0x7ffbfacd0000  0xf000  C:\Windows\SYSTEM32\websocket.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:38 AM
filter.dll  0x7ffbfacf0000  0x15000 C:\Windows\System32\inetsrv\filter.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:59 AM
isapi.dll   0x7ffbfad10000  0x24000 C:\Windows\System32\inetsrv\isapi.dll   Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:12:50 AM
compdyn.dll 0x7ffbfad40000  0xe000  C:\Windows\System32\inetsrv\compdyn.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:36 AM
winrnr.dll  0x7ffbfbeb0000  0xe000  C:\Windows\System32\winrnr.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:05 AM
napinsp.dll 0x7ffbfbec0000  0x16000 C:\Windows\system32\napinsp.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:28 AM
w3dt.dll    0x7ffbfccf0000  0x1f000 c:\windows\system32\inetsrv\w3dt.dll    Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:06:18 AM
iiscore.dll 0x7ffbfd740000  0x54000 C:\Windows\system32\inetsrv\iiscore.dll Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:10:40 AM
modrqflt.dll    0x7ffbfd7c0000  0xf000  C:\Windows\System32\inetsrv\modrqflt.dll    Microsoft Corporation   10.0.14393.2758 (rs1_release_1.190104-1904) 1/5/2019 7:13:10 AM
authanon.dll    0x7ffbfd7d0000  0x10000 C:\Windows\System32\inetsrv\authanon.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:39 AM
static.dll  0x7ffbfdc00000  0xf000  C:\Windows\System32\inetsrv\static.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:09 AM
protsup.dll 0x7ffbfdc10000  0xa000  C:\Windows\System32\inetsrv\protsup.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:17 AM
dirlist.dll 0x7ffbfdc20000  0xa000  C:\Windows\System32\inetsrv\dirlist.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:34 AM
compstat.dll    0x7ffbfdc50000  0x11000 C:\Windows\System32\inetsrv\compstat.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:17 AM
cachhttp.dll    0x7ffbfdc70000  0x11000 C:\Windows\System32\inetsrv\cachhttp.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:54 AM
defdoc.dll  0x7ffbfdc90000  0xa000  C:\Windows\System32\inetsrv\defdoc.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:44 AM
cachtokn.dll    0x7ffbfdca0000  0x9000  C:\Windows\System32\inetsrv\cachtokn.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:40 AM
cachfile.dll    0x7ffbfdcb0000  0xa000  C:\Windows\System32\inetsrv\cachfile.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:17 AM
cachuri.dll 0x7ffbfdcc0000  0x9000  C:\Windows\System32\inetsrv\cachuri.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:15 AM
loghttp.dll 0x7ffbfdcd0000  0xe000  C:\Windows\System32\inetsrv\loghttp.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:12 AM
w3wphost.dll    0x7ffbfdd60000  0x17000 c:\windows\system32\inetsrv\w3wphost.dll    Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:49:16 AM
HTTPAPI.dll 0x7ffc01090000  0xd000  C:\Windows\SYSTEM32\HTTPAPI.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/12/2017 6:20:24 AM
W3TP.dll    0x7ffc01470000  0xc000  c:\windows\system32\inetsrv\W3TP.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:13 AM
mlang.dll   0x7ffc03dc0000  0x3f000 C:\Windows\system32\mlang.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/23/2018 4:37:19 AM
VERSION.dll 0x7ffc049a0000  0xa000  C:\Windows\SYSTEM32\VERSION.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:11 AM
ktmw32.dll  0x7ffc04a50000  0xb000  C:\Windows\SYSTEM32\ktmw32.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:10:41 AM
nativerd.dll    0x7ffc04a60000  0x7c000 c:\windows\system32\inetsrv\nativerd.dll    Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:45:02 AM
iisutil.dll 0x7ffc04d50000  0x4d000 c:\windows\system32\inetsrv\iisutil.dll Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:46:54 AM
rasadhlp.dll    0x7ffc057b0000  0xa000  C:\Windows\System32\rasadhlp.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:58 AM
webio.dll   0x7ffc05a30000  0x90000 C:\Windows\SYSTEM32\webio.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/30/2018 9:33:32 PM
ondemandconnroutehelper.dll 0x7ffc05ae0000  0x15000 C:\Windows\SYSTEM32\ondemandconnroutehelper.dll Microsoft Corporation   10.0.14393.351 (rs1_release_inmarket.161014-1755)   10/15/2016 4:56:41 AM
XmlLite.dll 0x7ffc06bd0000  0x36000 C:\Windows\SYSTEM32\XmlLite.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:02 AM
fwpuclnt.dll    0x7ffc06eb0000  0x6a000 C:\Windows\System32\fwpuclnt.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:12:25 AM
WINNSI.DLL  0x7ffc082e0000  0xb000  C:\Windows\SYSTEM32\WINNSI.DLL  Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:49:47 AM
NLAapi.dll  0x7ffc088f0000  0x18000 C:\Windows\system32\NLAapi.dll  Microsoft Corporation   10.0.14393.2636 (rs1_release_1.181031-1836) 11/1/2018 5:25:13 AM
WINHTTP.dll 0x7ffc0a230000  0xcc000 C:\Windows\SYSTEM32\WINHTTP.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:03:27 AM
ntmarta.dll 0x7ffc0c3b0000  0x32000 C:\Windows\SYSTEM32\ntmarta.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:02 AM
rsaenh.dll  0x7ffc0cba0000  0x33000 C:\Windows\system32\rsaenh.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/23/2018 4:38:30 AM
DPAPI.DLL   0x7ffc0cbe0000  0xa000  C:\Windows\SYSTEM32\DPAPI.DLL   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:27:47 AM
IPHLPAPI.DLL    0x7ffc0cd20000  0x38000 C:\Windows\SYSTEM32\IPHLPAPI.DLL    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  6/12/2018 1:41:33 AM
DNSAPI.dll  0x7ffc0cd60000  0xa1000 C:\Windows\SYSTEM32\DNSAPI.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  6/26/2018 4:11:22 AM
mswsock.dll 0x7ffc0cfa0000  0x5c000 C:\Windows\System32\mswsock.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:22:57 AM
CRYPTSP.dll 0x7ffc0d140000  0x17000 C:\Windows\SYSTEM32\CRYPTSP.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:12 AM
CRYPTBASE.dll   0x7ffc0d160000  0xb000  C:\Windows\SYSTEM32\CRYPTBASE.dll   Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:28:23 AM
NTASN1.dll  0x7ffc0d1e0000  0x3b000 C:\Windows\SYSTEM32\NTASN1.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:32 AM
ncrypt.dll  0x7ffc0d220000  0x26000 C:\Windows\SYSTEM32\ncrypt.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/23/2018 4:38:40 AM
SspiCli.dll 0x7ffc0d340000  0x2c000 C:\Windows\SYSTEM32\SspiCli.dll Microsoft Corporation   10.0.14393.2580 (rs1_release_inmarket.181009-1745)  10/10/2018 8:48:34 AM
bcrypt.dll  0x7ffc0d610000  0x2b000 C:\Windows\SYSTEM32\bcrypt.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  6/8/2018 6:46:34 AM
profapi.dll 0x7ffc0d6d0000  0x14000 C:\Windows\System32\profapi.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:56 AM
powrprof.dll    0x7ffc0d6f0000  0x4c000 C:\Windows\System32\powrprof.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:21:19 AM
kernel.appcore.dll  0x7ffc0d740000  0xf000  C:\Windows\System32\kernel.appcore.dll  Microsoft Corporation   10.0.14393.2312 (rs1_release.180607-1919)   6/8/2018 6:48:18 AM
windows.storage.dll 0x7ffc0d760000  0x6d9000    C:\Windows\System32\windows.storage.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:17:36 AM
ucrtbase.dll    0x7ffc0de40000  0xf4000 C:\Windows\System32\ucrtbase.dll    Microsoft Corporation   10.0.14393.2848 (rs1_release.190305-1856)   3/6/2019 7:01:00 AM
win32u.dll  0x7ffc0df40000  0x1e000 C:\Windows\System32\win32u.dll  Microsoft Corporation   10.0.14393.51 (rs1_release_inmarket.160801-1836)    8/2/2016 9:21:20 AM
CRYPT32.dll 0x7ffc0e010000  0x1c9000    C:\Windows\System32\CRYPT32.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  4/3/2018 4:35:01 AM
gdi32full.dll   0x7ffc0e1e0000  0x181000    C:\Windows\System32\gdi32full.dll   Microsoft Corporation   10.0.14393.2791 (rs1_release.190205-1511)   2/6/2019 3:14:08 AM
shcore.dll  0x7ffc0e370000  0xa9000 C:\Windows\System32\shcore.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:04:31 AM
KERNELBASE.dll  0x7ffc0e420000  0x21d000    C:\Windows\System32\KERNELBASE.dll  Microsoft Corporation   10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)  3/6/2019 7:02:22 AM
msvcp_win.dll   0x7ffc0e6a0000  0x9c000 C:\Windows\System32\msvcp_win.dll   Microsoft Corporation   10.0.14393.2848 (rs1_release.190305-1856)   3/6/2019 7:00:46 AM
cfgmgr32.dll    0x7ffc0e740000  0x42000 C:\Windows\System32\cfgmgr32.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:26:04 AM
bcryptPrimitives.dll    0x7ffc0e790000  0x6a000 C:\Windows\System32\bcryptPrimitives.dll    Microsoft Corporation   10.0.14393.2457 (rs1_release_inmarket.180822-1743)  8/23/2018 4:37:31 AM
shlwapi.dll 0x7ffc0e800000  0x52000 C:\Windows\System32\shlwapi.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  7/16/2016 3:14:37 AM
advapi32.dll    0x7ffc0e860000  0xa2000 C:\Windows\System32\advapi32.dll    Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/30/2018 9:34:36 PM
GDI32.dll   0x7ffc0ed70000  0x34000 C:\Windows\System32\GDI32.dll   Microsoft Corporation   10.0.14393.2848 (rs1_release.190305-1856)   3/6/2019 7:07:25 AM
WS2_32.dll  0x7ffc0edb0000  0x6a000 C:\Windows\System32\WS2_32.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  9/15/2016 5:34:44 PM
KERNEL32.DLL    0x7ffc0ee20000  0xac000 C:\Windows\System32\KERNEL32.DLL    Microsoft Corporation   10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)  3/6/2019 7:16:30 AM
SHELL32.dll 0x7ffc0ef40000  0x1505000   C:\Windows\System32\SHELL32.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:12:44 AM
clbcatq.dll 0x7ffc105e0000  0x9f000 C:\Windows\System32\clbcatq.dll Microsoft Corporation   2001.12.10941.16384 (rs1_release.160715-1616)   7/16/2016 3:18:57 AM
NSI.dll 0x7ffc10960000  0x8000  C:\Windows\System32\NSI.dll Microsoft Corporation   10.0.14393.2339 (rs1_release_inmarket.180611-1502)  6/12/2018 1:49:58 AM
RPCRT4.dll  0x7ffc10970000  0x121000    C:\Windows\System32\RPCRT4.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  3/6/2019 7:01:18 AM
OLEAUT32.dll    0x7ffc10ba0000  0xbf000 C:\Windows\System32\OLEAUT32.dll    Microsoft Corporation   10.0.14393.2791 (rs1_release.190205-1511)   2/6/2019 3:11:09 AM
user32.dll  0x7ffc10d20000  0x165000    C:\Windows\System32\user32.dll  Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  4/28/2018 6:41:41 AM
sechost.dll 0x7ffc10ea0000  0x59000 C:\Windows\System32\sechost.dll Microsoft Corporation   10.0.14393.0 (rs1_release.160715-1616)  8/30/2018 9:34:58 PM
combase.dll 0x7ffc10f00000  0x2c5000    C:\Windows\System32\combase.dll Microsoft Corporation   10.0.14393.1198 (rs1_release_sec.170427-1353)   3/6/2019 7:04:41 AM
msvcrt.dll  0x7ffc111d0000  0x9e000 C:\Windows\System32\msvcrt.dll  Microsoft Corporation   7.0.14393.2457 (rs1_release_inmarket.180822-1743)   8/23/2018 4:39:07 AM
ntdll.dll   0x7ffc11270000  0x1d0000    C:\Windows\SYSTEM32\ntdll.dll   Microsoft Corporation   10.0.14393.2430 (rs1_release_inmarket_aim.180806-1810)  10/25/2018 4:09:08 AM

IRP_MJ_CREATE (CreateFile) - stack:

0   FLTMGR.SYS  FltDecodeParameters + 0x195a    0xfffff807bc1746ca  C:\Windows\System32\drivers\FLTMGR.SYS
1   FLTMGR.SYS  FltDecodeParameters + 0x1508    0xfffff807bc174278  C:\Windows\System32\drivers\FLTMGR.SYS
2   FLTMGR.SYS  FltQueryInformationFile + 0x6d1 0xfffff807bc1a4051  C:\Windows\System32\drivers\FLTMGR.SYS
3   ntoskrnl.exe    SeQueryInformationToken + 0x7d5f    0xfffff8008db1e84f  C:\Windows\system32\ntoskrnl.exe
4   ntoskrnl.exe    ObWaitForMultipleObjects + 0xf2f    0xfffff8008db3204f  C:\Windows\system32\ntoskrnl.exe
5   ntoskrnl.exe    ObOpenObjectByNameEx + 0x1dd    0xfffff8008db663dd  C:\Windows\system32\ntoskrnl.exe
6   ntoskrnl.exe    FsRtlFreeExtraCreateParameter + 0x24e   0xfffff8008db624ae  C:\Windows\system32\ntoskrnl.exe
7   ntoskrnl.exe    setjmpex + 0x68a3   0xfffff8008d7f9403  C:\Windows\system32\ntoskrnl.exe
8   ntdll.dll   NtQueryAttributesFile + 0x14    0x7ffc113162a4  C:\Windows\SYSTEM32\ntdll.dll
9   KERNELBASE.dll  GetFileAttributesW + 0x7e   0x7ffc0e44121e  C:\Windows\System32\KERNELBASE.dll
10  nativerd.dll    DllGetClassObject + 0x2778  0x7ffc04a63a48  c:\windows\system32\inetsrv\nativerd.dll
11  nativerd.dll    DllGetClassObject + 0x11f27 0x7ffc04a731f7  c:\windows\system32\inetsrv\nativerd.dll
12  nativerd.dll    DllGetClassObject + 0x142b0 0x7ffc04a75580  c:\windows\system32\inetsrv\nativerd.dll
13  iiscore.dll iiscore.dll + 0xa1a5    0x7ffbfd74a1a5  C:\Windows\system32\inetsrv\iiscore.dll
14  iiscore.dll iiscore.dll + 0xa535    0x7ffbfd74a535  C:\Windows\system32\inetsrv\iiscore.dll
15  iiscore.dll iiscore.dll + 0xb144    0x7ffbfd74b144  C:\Windows\system32\inetsrv\iiscore.dll
16  iiscore.dll iiscore.dll + 0x11bbf   0x7ffbfd751bbf  C:\Windows\system32\inetsrv\iiscore.dll
17  w3dt.dll    UlAtqSendEntityBody + 0xc5  0x7ffbfccf4c95  c:\windows\system32\inetsrv\w3dt.dll
18  W3TP.dll    THREAD_POOL::CreateThreadPool + 0x4f3   0x7ffc01471853  c:\windows\system32\inetsrv\W3TP.dll
19  W3TP.dll    THREAD_POOL::CreateThreadPool + 0x443   0x7ffc014717a3  c:\windows\system32\inetsrv\W3TP.dll
20  W3TP.dll    THREAD_POOL::CreateThreadPool + 0x399   0x7ffc014716f9  c:\windows\system32\inetsrv\W3TP.dll
21  KERNEL32.DLL    BaseThreadInitThunk + 0x14  0x7ffc0ee284d4  C:\Windows\System32\KERNEL32.DLL
22  ntdll.dll   RtlUserThreadStart + 0x21   0x7ffc112de851  C:\Windows\SYSTEM32\ntdll.dll
shirhatti commented 5 years ago

Paging @jkotalik

jkotalik commented 5 years ago

I'll take a look when I get the chance. Lot on my plate ATM.

shapeh commented 5 years ago

@jkotalik @shirhatti - thanks :)

I now see we also get this for .net 4.6.2 - but a lot less than for core. I suspect IIS problem:

FASTIO_NETWORK_QUERY_OPEN: result -> FAST IO DISALLOWED:

c:\windows\system32\inetsrv\w3wp.exe -ap "othersite.com" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipmab48c48d-d3de-41f3-a08b-3b60ebdcb599 -h "C:\inetpub\temp\apppools\othersite.com\othersite.com.config" -w "" -m 0 -t 20 -ta 0

IRP_MJ_CREATE: result -> PATH NOT FOUND:

c:\windows\system32\inetsrv\w3wp.exe -ap "othersite.com" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipmab48c48d-d3de-41f3-a08b-3b60ebdcb599 -h "C:\inetpub\temp\apppools\othersite.com\othersite.com.config" -w "" -m 0 -t 20 -ta 0

shapeh commented 5 years ago

A few more appcrashes happened tonight / not sure if this is related to OP. Running outofprocess 2.2.3

UPDATE: I found out that this happens when we share articles via our page to facebook (yes). E.g. when this request is done https://www.facebook.com/sharer/sharer.php?u=https://link-to-our-site.com - then fb will try and load og:-data like og:title, og:description, og:image etc. Then our site breaks down and below errors are found in event log, acnm.log etc. Perhaps some strange request is made from fb scraper to IIS which causes it to break down. We can initiate this error ourselves, simply by calling link above or testing via https://developers.facebook.com/tools/debug/sharing/?q=

On first request, fb scraper tells us nothing is found in og:, second request it will get og:title and other properties, 3rd request also og:image which is pointing to a cdn at cloudflare (which may or may not download image from same server as main site).

UPDATE Two: Same thing happens sharing an article on Skype... hummm.

This happens for OutOfProcess and InProcess. When using InProcess, app pool is automatically restarted, - OutOfProcess does not automatically restart app-pool.

Could it be service lifetime we have misconfigured or something?

services.Configure<EmailConfiguration>(Configuration.GetSection("EmailConfiguration"));
services.Configure<AppLanguages>(Configuration.GetSection("AppLanguages"));
services.Configure<DatabaseConfiguration>(Configuration.GetSection("ConnectionStrings"));
services.Configure<GeneralConfiguration>(Configuration.GetSection("GeneralConfiguration"));

services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
services.AddScoped<IViewRenderService, ViewRenderService>();
services.AddTransient<IUserStore<User>, UserStore>();
services.AddTransient<IRoleStore<IdentityRole>, UserStore>();
services.AddIdentity<User, IdentityRole>().AddDefaultTokenProviders();
services.AddAuthentication(CookieAuthenticationDefaults.AuthenticationScheme).AddCookie();

Some more logs to digest: Eventlog 1

Application: dotnet.exe
CoreCLR Version: 4.6.27414.5
Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FFBF981E8FB (00007FFBF9670000) with exit code c0000005.

Eventlog 2

Faulting application name: dotnet.exe, version: 2.2.27414.6, time stamp: 0x5c65fa1e
Faulting module name: coreclr.dll, version: 4.6.27414.5, time stamp: 0x5c65dbe5
Exception code: 0xc0000005
Fault offset: 0x00000000001ae8fb
Faulting process id: 0x18b8
Faulting application start time: 0x01d4dd02ef90d83a
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\coreclr.dll
Report Id: 8c56cda8-3708-4a59-a2c7-dd7a0a19bc79
Faulting package full name: 
Faulting package-relative application ID:

Eventlog 3 Failed to gracefully shutdown process '6328'.

Eventlog 4

Fault bucket 1991266777276657166, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: dotnet.exe
P2: 2.2.27414.6
P3: 5c65fa1e
P4: coreclr.dll
P5: 4.6.27414.5
P6: 5c65dbe5
P7: c0000005
P8: 00000000001ae8fb
P9: 
P10: 

Attached files:
\\?\C:\Windows\Temp\WER343C.tmp.appcompat.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER3556.tmp.WERInternalMetadata.xml
WERGenerationLog.txt

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_dotnet.exe_35417edee9a986d98e87288676a240e4c6b428b5_51fe8fab_2ca875e3

Analysis symbol: 
Rechecking for solution: 0
Report Id: 8c56cda8-3708-4a59-a2c7-dd7a0a19bc79
Report Status: 0
Hashed bucket: 0ba5f49fe2919c139ba26695709cae0e

Then from in ancm.log:

ANCM.log

[aspnetcorev2_outofprocess.dll] Event Log: 'Application '/LM/W3SVC/5/ROOT' started process '6328' successfully and process '6328' is listening on port '11609'.' 
End Event Log Message.
[aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening
[aspnetcorev2.dll] Stopping application '/LM/W3SVC/5/ROOT'
[aspnetcorev2_outofprocess.dll] Stopping file watcher thread
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:97 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:116 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:97 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:116 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:97 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:116 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:97 
[aspnetcorev2.dll] Failed HRESULT returned: 0x800704e7 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\aspnetcore\proxymodule.cpp:116 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x800705b4 at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\serverprocess.cpp:1309 
[aspnetcorev2_outofprocess.dll] Event Log: 'Failed to gracefully shutdown process '6328'.' 
End Event Log Message.
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x80072efe at c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\outofprocessrequesthandler\forwardinghandler.cpp:1447 
[aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 6444.. File Version: 12.2.18346.1. Description: IIS ASP.NET Core Module V2. Commit: 0f9ad16b096ca2535d77efd2ad27645449421b44.
[aspnetcorev2.dll] Loading request handler:  'C:\Program Files\IIS\Asp.Net Core Module\V2\12.2.18346.0\aspnetcorev2_outofprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_outofprocess.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\12.2.18346.0\aspnetcorev2_outofprocess.dll'. Process Id: 6444.. File Version: 12.2.18346.1. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 0f9ad16b096ca2535d77efd2ad27645449421b44.
[aspnetcorev2_outofprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\sites\www.pressport.com\'
[aspnetcorev2_outofprocess.dll] Starting file watcher thread
[aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::Terminate
[aspnetcorev2_outofprocess.dll] Event Log: 'Application '/LM/W3SVC/5/ROOT' started process '8276' successfully and process '8276' is listening on port '41801'.' 
End Event Log Message.

Report.wer

Version=1
EventType=APPCRASH
EventTime=131973315211288444
ReportType=2
Consent=1
UploadTime=131973316027848254
ReportIdentifier=288a70ef-48fb-11e9-a972-d80f2967c297
IntegratorReportIdentifier=8c56cda8-3708-4a59-a2c7-dd7a0a19bc79
NsAppName=dotnet.exe
AppSessionGuid=000018b8-0000-003a-3ad8-90ef02ddd401
TargetAppId=W:00068b63ae4062e2910113997ec582f2600700000904!0000c485f24d733d3555de41e01f3f77886cced6d6be!dotnet.exe
TargetAppVer=2019//02//14:23:30:38!26d35!dotnet.exe
BootId=4294967295
Response.BucketId=0ba5f49fe2919c139ba26695709cae0e
Response.BucketTable=4
Response.LegacyBucketId=1991266777276657166
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=dotnet.exe
Sig[1].Name=Application Version
Sig[1].Value=2.2.27414.6
Sig[2].Name=Application Timestamp
Sig[2].Value=5c65fa1e
Sig[3].Name=Fault Module Name
Sig[3].Value=coreclr.dll
Sig[4].Name=Fault Module Version
Sig[4].Value=4.6.27414.5
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=5c65dbe5
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=00000000001ae8fb
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.14393.2.0.0.400.8
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=96e0
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=96e011bddace2111ab66a964aa692892
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=d0c0
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=d0c0e2ac9fc384970b6d7b93465cf21b
UI[2]=C:\Program Files\dotnet\dotnet.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=.NET Core Host stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\Program Files\dotnet\dotnet.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL
LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\System32\ucrtbase.dll
LoadedModule[5]=C:\Program Files\dotnet\host\fxr\2.2.3\hostfxr.dll
LoadedModule[6]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\hostpolicy.dll
LoadedModule[7]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\coreclr.dll
LoadedModule[8]=C:\Windows\System32\ADVAPI32.dll
LoadedModule[9]=C:\Windows\System32\msvcrt.dll
LoadedModule[10]=C:\Windows\System32\sechost.dll
LoadedModule[11]=C:\Windows\System32\RPCRT4.dll
LoadedModule[12]=C:\Windows\System32\ole32.dll
LoadedModule[13]=C:\Windows\System32\combase.dll
LoadedModule[14]=C:\Windows\System32\bcryptPrimitives.dll
LoadedModule[15]=C:\Windows\System32\GDI32.dll
LoadedModule[16]=C:\Windows\System32\gdi32full.dll
LoadedModule[17]=C:\Windows\System32\USER32.dll
LoadedModule[18]=C:\Windows\System32\win32u.dll
LoadedModule[19]=C:\Windows\System32\OLEAUT32.dll
LoadedModule[20]=C:\Windows\System32\msvcp_win.dll
LoadedModule[21]=C:\Windows\System32\SHLWAPI.dll
LoadedModule[22]=C:\Windows\SYSTEM32\VERSION.dll
LoadedModule[23]=C:\Windows\SYSTEM32\bcrypt.dll
LoadedModule[24]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Private.CoreLib.dll
LoadedModule[25]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\clrjit.dll
LoadedModule[26]=C:\Windows\System32\kernel.appcore.dll
LoadedModule[27]=C:\inetpub\sites\www.example.com\example.dll
LoadedModule[28]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.dll
LoadedModule[29]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Hosting.Abstractions.dll
LoadedModule[30]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\netstandard.dll
LoadedModule[31]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Hosting.dll
LoadedModule[32]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.dll
LoadedModule[33]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.Abstractions.dll
LoadedModule[34]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.DependencyInjection.Abstractions.dll
LoadedModule[35]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Hosting.Abstractions.dll
LoadedModule[36]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.FileProviders.Abstractions.dll
LoadedModule[37]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Collections.dll
LoadedModule[38]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.dll
LoadedModule[39]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.EnvironmentVariables.dll
LoadedModule[40]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Primitives.dll
LoadedModule[41]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.Tasks.dll
LoadedModule[42]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.Extensions.dll
LoadedModule[43]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Linq.dll
LoadedModule[44]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.IO.FileSystem.dll
LoadedModule[45]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.CommandLine.dll
LoadedModule[46]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Server.Kestrel.Core.dll
LoadedModule[47]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.dll
LoadedModule[48]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Server.Kestrel.dll
LoadedModule[49]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Logging.dll
LoadedModule[50]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Server.IIS.dll
LoadedModule[51]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.InteropServices.RuntimeInformation.dll
LoadedModule[52]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Server.IISIntegration.dll
LoadedModule[53]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.DependencyInjection.dll
LoadedModule[54]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.ComponentModel.dll
LoadedModule[55]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Logging.Abstractions.dll
LoadedModule[56]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Http.Features.dll
LoadedModule[57]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Diagnostics.DiagnosticSource.dll
LoadedModule[58]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Http.Abstractions.dll
LoadedModule[59]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Http.dll
LoadedModule[60]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.ObjectPool.dll
LoadedModule[61]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Core.dll
LoadedModule[62]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Razor.Runtime.dll
LoadedModule[63]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Resources.ResourceManager.dll
LoadedModule[64]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.FileProviders.Physical.dll
LoadedModule[65]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.FileExtensions.dll
LoadedModule[66]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.Json.dll
LoadedModule[67]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.dll
LoadedModule[68]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.IO.FileSystem.Watcher.dll
LoadedModule[69]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.ComponentModel.Primitives.dll
LoadedModule[70]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Collections.Concurrent.dll
LoadedModule[71]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.FileSystemGlobbing.dll
LoadedModule[72]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.Timer.dll
LoadedModule[73]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Diagnostics.Tracing.dll
LoadedModule[74]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.Overlapped.dll
LoadedModule[75]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Newtonsoft.Json.dll
LoadedModule[76]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Linq.Expressions.dll
LoadedModule[77]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.ComponentModel.TypeConverter.dll
LoadedModule[78]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.ObjectModel.dll
LoadedModule[79]=C:\Windows\System32\clbcatq.dll
LoadedModule[80]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.Numerics.dll
LoadedModule[81]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.InteropServices.dll
LoadedModule[82]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Options.dll
LoadedModule[83]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll
LoadedModule[84]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Hosting.Server.Abstractions.dll
LoadedModule[85]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Logging.Configuration.dll
LoadedModule[86]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Options.ConfigurationExtensions.dll
LoadedModule[87]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Logging.Console.dll
LoadedModule[88]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Logging.Debug.dll
LoadedModule[89]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Logging.EventSource.dll
LoadedModule[90]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.HostFiltering.dll
LoadedModule[91]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.HttpOverrides.dll
LoadedModule[92]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Authentication.Core.dll
LoadedModule[93]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Authentication.Abstractions.dll
LoadedModule[94]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Security.Claims.dll
LoadedModule[95]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.CookiePolicy.dll
LoadedModule[96]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Localization.dll
LoadedModule[97]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.ResponseCaching.dll
LoadedModule[98]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.dll
LoadedModule[99]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Localization.dll
LoadedModule[100]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Razor.dll
LoadedModule[101]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.DataAnnotations.dll
LoadedModule[102]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Identity.Core.dll
LoadedModule[103]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Identity.Stores.dll
LoadedModule[104]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Identity.dll
LoadedModule[105]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Authentication.dll
LoadedModule[106]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Authentication.Cookies.dll
LoadedModule[107]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Routing.dll
LoadedModule[108]=C:\inetpub\sites\www.example.com\WebMarkupMin.AspNetCore2.dll
LoadedModule[109]=C:\inetpub\sites\www.example.com\WebMarkupMin.AspNet.Common.dll
LoadedModule[110]=C:\inetpub\sites\www.example.com\WebMarkupMin.Core.dll
LoadedModule[111]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.ResponseCompression.dll
LoadedModule[112]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Diagnostics.HealthChecks.dll
LoadedModule[113]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Diagnostics.HealthChecks.Abstractions.dll
LoadedModule[114]=C:\inetpub\sites\www.example.com\HealthChecks.SqlServer.dll
LoadedModule[115]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.HttpsPolicy.dll
LoadedModule[116]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Localization.Abstractions.dll
LoadedModule[117]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Caching.Memory.dll
LoadedModule[118]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Caching.Abstractions.dll
LoadedModule[119]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.DependencyModel.dll
LoadedModule[120]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Reflection.dll
LoadedModule[121]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.AppContext.dll
LoadedModule[122]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.IO.FileSystem.Primitives.dll
LoadedModule[123]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.IO.dll
LoadedModule[124]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Private.Uri.dll
LoadedModule[125]=C:\inetpub\sites\www.example.com\example.Views.dll
LoadedModule[126]=C:\inetpub\sites\www.example.com\MiniProfiler.AspNetCore.Mvc.dll
LoadedModule[127]=C:\inetpub\sites\www.example.com\SimpleMvcSitemap.dll
LoadedModule[128]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Identity.UI.Views.V3.dll
LoadedModule[129]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.RazorPages.dll
LoadedModule[130]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Identity.UI.Views.V4.dll
LoadedModule[131]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Identity.UI.dll
LoadedModule[132]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.SpaServices.dll
LoadedModule[133]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.SpaServices.Extensions.dll
LoadedModule[134]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Routing.Abstractions.dll
LoadedModule[135]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Abstractions.dll
LoadedModule[136]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Buffers.dll
LoadedModule[137]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.ApiExplorer.dll
LoadedModule[138]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Authorization.dll
LoadedModule[139]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Authorization.Policy.dll
LoadedModule[140]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.TagHelpers.dll
LoadedModule[141]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Razor.dll
LoadedModule[142]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.ViewFeatures.dll
LoadedModule[143]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Formatters.Json.dll
LoadedModule[144]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.DataProtection.dll
LoadedModule[145]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.DataProtection.Abstractions.dll
LoadedModule[146]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Cryptography.Internal.dll
LoadedModule[147]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Antiforgery.dll
LoadedModule[148]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.WebEncoders.dll
LoadedModule[149]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\System.Text.Encodings.Web.dll
LoadedModule[150]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.CodeAnalysis.Razor.dll
LoadedModule[151]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Razor.Language.dll
LoadedModule[152]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.CodeAnalysis.dll
LoadedModule[153]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Razor.Extensions.dll
LoadedModule[154]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Mvc.Cors.dll
LoadedModule[155]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Cors.dll
LoadedModule[156]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Extensions.Configuration.Binder.dll
LoadedModule[157]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Console.dll
LoadedModule[158]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.Thread.dll
LoadedModule[159]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Memory.dll
LoadedModule[160]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Connections.Abstractions.dll
LoadedModule[161]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Net.Primitives.dll
LoadedModule[162]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\System.IO.Pipelines.dll
LoadedModule[163]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\Microsoft.Win32.Registry.dll
LoadedModule[164]=C:\Windows\System32\shell32.dll
LoadedModule[165]=C:\Windows\System32\cfgmgr32.dll
LoadedModule[166]=C:\Windows\System32\windows.storage.dll
LoadedModule[167]=C:\Windows\System32\powrprof.dll
LoadedModule[168]=C:\Windows\System32\shcore.dll
LoadedModule[169]=C:\Windows\System32\profapi.dll
LoadedModule[170]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Security.Principal.Windows.dll
LoadedModule[171]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Security.Principal.dll
LoadedModule[172]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Xml.XDocument.dll
LoadedModule[173]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Private.Xml.Linq.dll
LoadedModule[174]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Private.Xml.dll
LoadedModule[175]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Security.Cryptography.Algorithms.dll
LoadedModule[176]=C:\Windows\System32\crypt32.dll
LoadedModule[177]=C:\Windows\System32\MSASN1.dll
LoadedModule[178]=C:\Windows\SYSTEM32\DPAPI.DLL
LoadedModule[179]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[180]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Text.Encoding.Extensions.dll
LoadedModule[181]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Text.RegularExpressions.dll
LoadedModule[182]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Reflection.Emit.ILGeneration.dll
LoadedModule[183]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Reflection.Emit.Lightweight.dll
LoadedModule[184]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Reflection.Primitives.dll
LoadedModule[185]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Diagnostics.dll
LoadedModule[186]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Localization.dll
LoadedModule[187]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.StaticFiles.dll
LoadedModule[188]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Diagnostics.HealthChecks.dll
LoadedModule[189]=C:\inetpub\sites\www.example.com\MiniProfiler.AspNetCore.dll
LoadedModule[190]=C:\inetpub\sites\www.example.com\MiniProfiler.Shared.dll
LoadedModule[191]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.JsonPatch.dll
LoadedModule[192]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.Net.Http.Headers.dll
LoadedModule[193]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Collections.Specialized.dll
LoadedModule[194]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Drawing.Primitives.dll
LoadedModule[195]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Collections.Immutable.dll
LoadedModule[196]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Security.Cryptography.Primitives.dll
LoadedModule[197]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.Serialization.Formatters.dll
LoadedModule[198]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.IO.Compression.dll
LoadedModule[199]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Net.Sockets.dll
LoadedModule[200]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Net.NameResolution.dll
LoadedModule[201]=C:\Windows\System32\ws2_32.dll
LoadedModule[202]=C:\Windows\system32\napinsp.dll
LoadedModule[203]=C:\Windows\System32\mswsock.dll
LoadedModule[204]=C:\Windows\SYSTEM32\DNSAPI.dll
LoadedModule[205]=C:\Windows\System32\NSI.dll
LoadedModule[206]=C:\Windows\SYSTEM32\IPHLPAPI.DLL
LoadedModule[207]=C:\Windows\System32\winrnr.dll
LoadedModule[208]=C:\Windows\system32\NLAapi.dll
LoadedModule[209]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.ThreadPool.dll
LoadedModule[210]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Numerics.Vectors.dll
LoadedModule[211]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\System.Runtime.CompilerServices.Unsafe.dll
LoadedModule[212]=C:\Windows\System32\Normaliz.dll
LoadedModule[213]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Security.Cryptography.X509Certificates.dll
LoadedModule[214]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Diagnostics.Debug.dll
LoadedModule[215]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Http.Extensions.dll
LoadedModule[216]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.ResponseCaching.Abstractions.dll
LoadedModule[217]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.ComponentModel.Annotations.dll
LoadedModule[218]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Data.Common.dll
LoadedModule[219]=C:\inetpub\sites\www.example.com\Dapper.dll
LoadedModule[220]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Diagnostics.Tools.dll
LoadedModule[221]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Html.Abstractions.dll
LoadedModule[222]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Text.Encoding.dll
LoadedModule[223]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Xml.XmlSerializer.dll
LoadedModule[224]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.CodeAnalysis.CSharp.dll
LoadedModule[225]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.Diagnostics.Abstractions.dll
LoadedModule[226]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Threading.Tasks.Extensions.dll
LoadedModule[227]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\Microsoft.AspNetCore.WebUtilities.dll
LoadedModule[228]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\Microsoft.CSharp.dll
LoadedModule[229]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\System.Data.SqlClient.dll
LoadedModule[230]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Transactions.Local.dll
LoadedModule[231]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Xml.XPath.XDocument.dll
LoadedModule[232]=C:\inetpub\sites\www.example.com\runtimes\win\lib\netcoreapp2.0\System.Runtime.Caching.dll
LoadedModule[233]=C:\inetpub\sites\www.example.com\System.Configuration.ConfigurationManager.dll
LoadedModule[234]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Xml.ReaderWriter.dll
LoadedModule[235]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Net.WebClient.dll
LoadedModule[236]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\sni.dll
LoadedModule[237]=C:\Windows\system32\secur32.dll
LoadedModule[238]=C:\Windows\SYSTEM32\SSPICLI.DLL
LoadedModule[239]=C:\Windows\system32\schannel.DLL
LoadedModule[240]=C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\2.2.3\System.Text.Encoding.CodePages.dll
LoadedModule[241]=C:\Windows\SYSTEM32\mskeyprotect.dll
LoadedModule[242]=C:\Windows\SYSTEM32\ncrypt.dll
LoadedModule[243]=C:\Windows\SYSTEM32\NTASN1.dll
LoadedModule[244]=C:\Windows\system32\ncryptsslp.dll
LoadedModule[245]=C:\inetpub\sites\www.example.com\da\example.resources.dll
LoadedModule[246]=C:\inetpub\sites\www.example.com\HtmlAgilityPack.dll
LoadedModule[247]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\clrcompression.dll
LoadedModule[248]=C:\inetpub\sites\www.example.com\es\example.resources.dll
LoadedModule[249]=C:\inetpub\sites\www.example.com\sv\example.resources.dll
LoadedModule[250]=C:\inetpub\sites\www.example.com\no\example.resources.dll
LoadedModule[251]=C:\inetpub\sites\www.example.com\de\example.resources.dll
LoadedModule[252]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Runtime.Serialization.Primitives.dll
LoadedModule[253]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\System.Diagnostics.TraceSource.dll
LoadedModule[254]=C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\Microsoft.Win32.Primitives.dll
State[0].Key=Transport.DoneStage1
State[0].Value=1
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=.NET Core Host
AppPath=C:\Program Files\dotnet\dotnet.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=B1D0B431B2B5D2C1D2138CD17AF91470
MetadataHash=418882279
shapeh commented 5 years ago

Managed to get a dump file with procdump. Anybody interested in this? 437mb.


C:\Users\Administrator\Desktop\Procdump>procdump64 -ma -e -t dotnet.exe

ProcDump v9.0 - Sysinternals process dump utility
Copyright (C) 2009-2017 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

Process:               dotnet.exe (704)
Process image:         C:\Program Files\dotnet\dotnet.exe
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     n/a
Hung window check:     Disabled
Log debug strings:     Disabled
Exception monitor:     Unhandled
Exception filter:      [Includes]
                       *
                       [Excludes]
Terminate monitor:     Enabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       1
Dump folder:           C:\Users\Administrator\Desktop\Procdump\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS
Queue to WER:          Disabled
Kill after dump:       Disabled

[16:57:50] Exception: E0434352.CLR
[16:57:50] Exception: E0434352.CLR
[16:57:51] Exception: E0434352.CLR
[16:57:51] Exception: E0434352.CLR
[16:58:05] Exception: C0000005.ACCESS_VIOLATION
[16:58:05] Dump 1 initiated: C:\Users\Administrator\Desktop\Procdump\dotnet.exe_190318_165805.dmp
[16:58:05] Dump 1 writing: Estimated dump file size is 436 MB.
[16:58:06] Dump 1 complete: 437 MB written in 1.7 seconds
[16:58:07] The process has exited.
[16:58:07] Dump count reached.
shirhatti commented 5 years ago

@shapeh I can take a look. Can you email me a link at [firstname].[lastname][at]microsoft.com ?

shapeh commented 5 years ago

@shirhatti - sent to your email as a onedrive link :) Thanks for looking into this.

shapeh commented 5 years ago

@shirhatti We have obtained more dumps if needed - this time we used 'EnableDumps PowerShell script' from here: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/troubleshoot?view=aspnetcore-2.2#create-a-dump

We have registered this error probably 40 times today.

jkotalik commented 5 years ago

@shapeh did this behavior reproduce on the original AspNetCoreModule. Ex: target the AspNetCoreModule as the module name in your web.config?

From what I read, I don't see why this would be an IIS issue. Feel free to send me the dumps too at the email listed on my github profile, because I'm curious.

shapeh commented 5 years ago

@jkotalik - we have not tried with AspNetCoreModule - but are trying now to see what will happen. I have just sent you an email with 3 dumps - 2 from PS and 1from procdump64 + analysis report.

It could a misconfiguration somewhere on our side but I don't really know. For the CreateFile error, I am not even sure they are related to the dumps I am sending.

jkotalik commented 5 years ago

For the CreateFile error, I am not even sure they are related to the dumps I am sending.

I don't believe they are. The dumps you are sending just have the dotnet process throwing an unhandled exception.

shapeh commented 5 years ago

Okay I understand.

We have switched to AspNetCoreModule now and have been running for ~2 hours with no problems so far. In contract, we also do not have a lot of traffic on the site these hours.

Do you think the exception is due to a mis configuration in our code or? It is kind of hard to track it down as the log files are pretty limited.

shapeh commented 5 years ago

Update: we still get the exceptions with AspNetCoreModule:

Faulting application name: dotnet.exe, version: 2.2.27414.6, time stamp: 0x5c65fa1e
Faulting module name: coreclr.dll, version: 4.6.27414.5, time stamp: 0x5c65dbe5
Exception code: 0xc0000005
Fault offset: 0x00000000001ae8fb
Faulting process id: 0xc10
Faulting application start time: 0x01d4de246a0feef3
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.3\coreclr.dll
Report Id: 60527bc9-ded3-490d-9138-53b9e59f03d0
Faulting package full name: 
Faulting package-relative application ID:
jkotalik commented 5 years ago

From the information I gathered, it seems like you have an error with your app and not with ANCM/IIS. I looked at the dumps and it shows an exception being thrown on a background thread. Look for any instances of async void. Also, note that the exception code 0xc00000005 (Access Violation) can be the same as a NullReferenceException.

shapeh commented 5 years ago

@jkotalik - we looked all code through. We have no async void. All asyncs´ are 

async Task<IActionResult> 
async Task<some model>
async Task<bool>
async Task<string>
async Task<IViewComponentResult>
and a few async Task<object[]>

None of the exceptions should be NRE's. My question is - why are they only being triggered when facebook sends a request or Skype sends a request? Also, it seems we can trigger those exceptions ourselves, i.e. click on the facebook-share link / Skype sharing, but only when running with AspNetCoreModule2 - same problem does not happen with AspNetCoreModule.

Are you aware of any other methods / logging we can use to debug the app? - we are really struggling with this stuff.

Some routing problem maybe?

jkotalik commented 5 years ago

Idea from out of left field: can you try to see what happens when you turn stdoutLogEnabled to false and rerun using ANCMV2?

shapeh commented 5 years ago

@jkotalik - tried both suggestions. CreateFile and error still happens - I have just sent you another dump file in another bug report (https://github.com/aspnet/AspNetCore/issues/6097)

ghost commented 4 years ago

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!