startMode="AlwaysRunning" for inProcess ANCM

[shirhatti]

There was an earlier issue asking for this (https://github.com/aspnet/AspNetCoreModule/issues/29) which we declined to do when ANCM was only out of process.

At the minimum, we need to ensure ANCM doesn't break with AlwaysRunning for 2,1. I suspect it might just work, but @jkotalik and @pan-wang know better.

[pan-wang]

AlwaysRunning just let WAS start w3wp.exe. Customer's true ask is about "Application_Start" or a "Startup" class in asp.Net core. I don't think asp.net core emit such notification. @davidfowl @Tratcher can you confirm it?

[asbjornu]

I'm with @pan-wang in that the requirement here might be two or even threefold. Solving the IIS worker process recycle mechanism isn't a real solution, at least not to me. Microsoft has managed to make this very complicated and hard to understand by having three different related and similar features for "keeping an IIS application warm" (which I believe is what most developers want):

1. Implementing the IProcessHostPreloadClient interface and registering it within the serviceAutoStartProviders section of IIS.
2. Configure the AutoStart feature.
3. Install and configure the applicationInitialization IIS module.

I think the first requires the second point, but I have no idea how they interact with the third. Why this has to be so difficult is impossible for me to understand, but I believe the feature request is as simple as: Please give us a way to control how our ASP.NET Core web applications are recycled and when they are, make IIS perform an HTTP request to it so the application can warm itself up.

[tuespetre]

@shirhatti @asbjornu my original request was basically this: when the application pool starts I want the app to also start. I don’t want it to wait for a request to start.

[sergioadh]

I participated in the previous thread and I was asking for what @tuespetre said as well

[shirhatti]

my original request was basically this: when the application pool starts I want the app to also start. I don’t want it to wait for a request to start.

Yup, we're on the same page here. Forgive my rather ambiguous issue title

[Looooooka]

https://blogs.msdn.microsoft.com/benjaminperkins/2014/01/07/configure-the-iis-application-initialization-module/ this works for me

[jkotalik]

I think we need to keep the current AlwaysRunning functionality the same as before. If we want to support "when the application pool starts I want the app to also start," we should design a different feature. For example, @BillHiebert wanted to use "AlwaysRunning" to allow VS to debug attach to w3wp.exe before the first request comes in.

[shirhatti]

I disagree. The semantic meaning of the feature is not just to start w3wp.exe, but rather start your application as well. This is what happens with a system.web app today. Just because we got we never got around to doing it in ANCM thus far, we don't have to preserve that status quo.

Designing it is a new feature is not a viable option since we can't modify IIS schema to add another attribute to the AppPool.

[jkotalik]

Will this be any different from the app warmup module https://blogs.msdn.microsoft.com/bryang/2011/04/29/iis-application-warm-up-module/ ?

[rockerinthelocker]

This thread is somewhat confusing because IIS Application Initialization already works well with ASP.NET Core (via startMode="AlwaysRunning" and preloadEnabled="true" settings) to 'warm up' an ASP.NET Core application (i.e. via 'Configure' method). Can anyone shed some light on this? Thank you!

[sergioadh]

AlwaysRunning and preloadEnabled control the application pool but not the ASP.NET Core app, which shuts down somehow. Even though the app pool is "warmed-up" the first requests will take a long time since the ASP.NET Core app is not "warmed-up".

There were previous issues and this has been moved around but you can read them:

https://github.com/aspnet/AspNetCoreModule/issues/29

https://github.com/aspnet/IISIntegration/issues/161

[tuespetre]

@jkotalik @rockerinthelocker:

The difference between this proposed feature and using the warmup/initialization module is that this proposed feature would not require a 'sentinel' web request to be issued to kick off the application. Additionally, when an application running under ANCM 'crashes', ANCM will simply wait until another request is issued to start up the application again. This is fundamentally disconnected from the application pool lifecycle, and thus the initialization module.

[rockerinthelocker]

@sergioadh , Well, it's up to the application developer to implement 'warm up' tasks to be executed on application start. There is no magic the ASP.NET Core module can do for developers.

@tuespetre , If it were possible to implement such a feature, no hosting provider would install the ASP.NET Core module on shared servers. Just imagine thousands of sites hosted on a server would 'warm up' on each IIS start/reset/configuration change.

[Looooooka]

I really don't see what the problem is here. The module literally allows you to set a start url which is called when the pool is started. All you need to do is handle it and preload whatever you want.

[sergioadh]

That's ok and good but the Kestrel server goes down even though the ApplicationPool is up, so when the app pool recycles it will send the request and we can get everything up and running but then Kestrel goes down and if a request comes in it takes a while to respond.

[pan-wang]

@sergioadh I got your point now. This is doable. ANCM does listen on the process exit of Kestrel server. If the backend is down, ANCM can immediately start another one. We need a configure flag to allow user to tune the behavior. This can be feature for next release.

[rockerinthelocker]

@pan-wang @jkotalik @Tratcher @muratg @coolcsh @davidfowl @DamianEdwards @halter73 , Simply restarting an application that might be unhealthy is certainly not the proper course of action; and probably not the job of the ASP.NET Core module anyway.

However, looking at how things are handled during application pool start/recycle/stop and site (application) start/restart/stop, it seems they are not handled properly.

Here is a simple setup (requires IIS Application Initialization) to reproduce the results shown below (replace 'C:\results.txt' with a path the application has write access to; and set startMode="AlwaysRunning" on the application pool and preloadEnabled="true" on the site's default application in applicationHost.config).

Startup.cs:

public void Configure(IApplicationBuilder app, IHostingEnvironment env, IApplicationLifetime appLife)
{
    System.IO.File.AppendAllText(@"C:\results.txt", $"Start executing Configure()...{System.Environment.NewLine}");
    System.IO.File.AppendAllText(@"C:\results.txt", $"Register IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped...{System.Environment.NewLine}");

    appLife.ApplicationStarted.Register(ApplicationStarted);
    appLife.ApplicationStopping.Register(ApplicationStopping);
    appLife.ApplicationStopped.Register(ApplicationStopped);

    app.UseMvc(routes =>
    {
        routes.MapRoute(
        name: "default",
        template: "{controller=Home}/{action=Index}/{id?}");
    });

    System.IO.File.AppendAllText(@"C:\results.txt", $"Finished executing Configure()...{System.Environment.NewLine}");
}

private void ApplicationStarted()
{
    System.IO.File.AppendAllText(@"C:\results.txt", $"Executing ApplicationStarted()...{System.Environment.NewLine}");
}
private void ApplicationStopping()
{
    System.IO.File.AppendAllText(@"C:\results.txt", $"Executing ApplicationStopping()...{System.Environment.NewLine}");
}
private void ApplicationStopped()
{
    System.IO.File.AppendAllText(@"C:\results.txt", $"Executing ApplicationStopped()...{System.Environment.NewLine}");
}

HomeController.cs

public class HomeController : Controller
{
    public IActionResult Index()
    {
        System.IO.File.AppendAllText(@"C:\results.txt", $"Executing HomeController.Index()...{System.Environment.NewLine}");

        return View();
    }
}

1) Application Pool Start

Start executing Configure()... Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped... Finished executing Configure()... Executing ApplicationStarted() event handler... Executing HomeController.Index()...

2) Application Pool Recycle

Start executing Configure()... Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped... Finished executing Configure()... Executing ApplicationStarted() event handler... Executing HomeController.Index()... Executing ApplicationStopping() event handler... Executing ApplicationStopped() event handler...

3) Request After Application Pool Recycle

Executing HomeController.Index()...

4) Application Pool Stop

Executing ApplicationStopping() event handler... Executing ApplicationStopped() event handler...

5) Application Pool Start

Start executing Configure()... Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped... Finished executing Configure()... Executing ApplicationStarted() event handler... Executing HomeController.Index()...

6) Site Stop

Executing ApplicationStopping() event handler... Executing ApplicationStopped() event handler...

7) Site Start

Nothing logged. Application is not started.

Conclusion: 1) ApplicationStopping and ApplicationStopped being raised on application pool recycle is certainly a bug. 2) Starting/restarting the site in IIS (unloads the application) should behave the same way as if the application pool was started (i.e. starting up the application).

[davidfowl]

ApplicationStopping and ApplicationStopped being raised on application pool recycle is certainly a bug.

I'm confused by this. A recycle is a stop then a start. Can you clarify what you mean here? What are you expecting?

Starting/restarting the site in IIS (unloads the application) should behave the same way as if the application pool was started (i.e. starting up the application).

Site starting in IIS AFAIK is just removing and adding bindings. There's no process activity associated with sites, just application pools (since those are the processes).

[rockerinthelocker]

@davidfowl , Sure, the proper order of 'events' logged should be

2. Application Pool Recycle

Executing ApplicationStopping() event handler... Executing ApplicationStopped() event handler... Start executing Configure()... Register event handlers for IApplicationLifetime.ApplicationStarted, IApplicationLifetime.ApplicationStopping, IApplicationLifetime.ApplicationStopped... Finished executing Configure()... Executing ApplicationStarted() event handler... Executing HomeController.Index()...

However, ApplicationStopping/ApplicationStopped are raised after the new application pool is started.

Actually, when stopping a site, IIS sets the 'serverAutoStart' attribute to 'false'. Anyway, ANCM unloads the application if the site is stopped (i.e. during restart) so it should reload the application if the site is started again. Because site start/stop triggers a configuration change event, ANCM may subscribe to that event.

[KLuuKer]

@davidfowl i think @rockerinthelocker is talking about (or not aware of) overlapping recycles

[davidfowl]

@rockerinthelocker can you update your logs to print the process ID prefixed on each of the lines?

[rockerinthelocker]

@davidfowl , Thanks for the heads up! Indeed, the overlapped application pool recycle caused the confusion here. So, just the issue with site start is left.

[thegoodnerd]

@rockerinthelocker I'm observing exactly the same behavior and have this far been unable to find a way around this.

[dotnetcanuck]

@rockerinthelocker @winretri I found my way here after having the exact same issues with an application at my company. We're seeing startup times (on initial requests) north of 30 seconds, with subsequent requests being processed at the speed of light.

I'm sorry, but that is completely ridiculous for a web application.

@sergioadh @Looooooka Can either of you elaborate on how I can specify a 'startup' URL for the AspNetCoreModule to handle? I can't find anything in the configuration reference.

[orionstudt]

@dotnetcanuck I don't think a Startup URL will solve the problem for you, because as @sergioadh pointed out - the Startup URL will be hit when the Application Pool starts, but then Kestrel will go down when it is idle.

My short term resolution is a separate console application that regularly pings the app to prevent Kestrel from going down.. but that is less than ideal. Hence the feature request.

[dotnetcanuck]

console application that regularly pings the app to prevent Kestrel from going down.. but that is less than ideal.

@orionstudt I'll say! We're looking at doing something similar to work around the issue for now.

Really hoping that Microsoft can implement this feature though. 😞

[dotnetcanuck]

@orionstudt How often (ballpark) would you say that Kestrel takes itself down?

Still looking at implementing a console app or health check, but in our testing, we're finding that if the API gets hit several times in quick succession, requests will suddenly start getting queued, and the application gets hung/paused for about 40 seconds until API calls begin getting responses.

Obviously, I have more investigation to do on that, and I don't mean to hijack the thread. But I guess my main thought there is, is Kestrel shutting itself down suddenly, or is there something else going on?

[rockerinthelocker]

@dotnetcanuck , The Configure method in Startup.cs is certainly a good place to run 'warm up' tasks. In order to let IIS send a user request to the application on start/recycle, set

startMode="AlwaysRunning" on the application pool, serverAutoStart="true" on the site (set by default), and preloadEnabled="true" on the application.

References: startMode: https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/index serverAutoStart: https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/site/ preloadEnabled: https://docs.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/site/application/

Note, however, that when publishing the application (i.e. via Visual Studio), IIS won't send the user request and you would have to recycle the application pool.

[MovGP0]

In Azure, there is the Always on option that just works and don't require any hacks. Unfortunately, it isn't as trivial for on-premise applications.

In IIS, I've set Start Mode to AlwaysRunning on the Application Pool and Preload Enabled to True in IIS. Still, the ASP.NET Core (Kestrel hosted) web application goes to sleep, killing the nightly Hangfire Background jobs with it.

So the best workaround seems to be a polling service that does regular health checks. Unfortunately, that is not an option everywhere. So you might need to have a Task Scheduler (Windows) or cron job (Linux/Unix) to implement a simple polling service.

Reference: Polling Service with Task Scheduler

As a workaround, I'm using Task Scheduler to create a task that runs every minute and polls the health check endpoint of my service.

• Open Task Scheduler
• Create Task...
• On General, enter a Name and use Change User Group... to define the user permission. Set Run whether user is logged on or not.
• On Triggers, use New... to create a new trigger.
  • Set Begin the task: to At Startup
  • Set Repeat task every to 5 minutes
  • Set for a duration of: to Indefinitely
  • Set enabled
  • OK
• On Actions, use New... to create a new action
  • Set Action: to Start a program
  • Set Program/script to powershell.exe
  • Set Add arguments (optional) to 'curl "https://mywebsite/hc"' (change the URL to whatever endpoint you need)
  • OK
• OK
• Enter the User password as needed.

Make sure that the user you choose has the proper privileges.

Set User rights for batch job

• Open gpedit.msc and navigate to Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment.
• Right-click Log on as a batch job and select Properties.
• Add User or Group...
• enter the user name in the form of DOMAIN\USERNAME and click OK
• OK

You may want to test-run the service at this point. If everything works, you can restart the computer. Check the Task History to ensure everything works properly.

[Looooooka]

Like I wrote before...always running. Plus under configuration editor(not site but the main icon set under iis manager) click on system.applicationHost/sites. You can enable autostart there. Now if you want/need a timer just start a timer when the app goes live and use a webclient or httpnetclient to open an url every x seconds. On windows in the past you could literaly put an object inside the iis process which would stay alive when iis would do the apppool cycling every 20 minutes. But this works just fine as well since kestrel will be restarted in case errors start popping up. Been using this technique in production since forever and we know exactly when something dies/doesn't start.

[MattComb]

I'd like to add my name to the long list of people effected by this issue. Without the ability to auto initialize/start the kestrel dot net core app when the iis app pool thread is started/refreshed (As opposed to waiting for an inbound http request), any web app that contains a background task basically becomes untenable.

e.g. If you had a micro service which had a udp service registered within the dotnet core app, it would not receive any messages until an inbound http request occurred

This is a critical fault in my opinion, please resolve this asap.

I suspect maybe an additional running mode "AlwaysRunningWithKeepAlive" to extend the AlwaysRunning setting to include the external kestrel service or an additional KeepAwake feature may be required.

[shirhatti]

Closing this issue since it's become a catch-all.

For the most requested scenario in this thread which is to have your backend dotnet.exe process started when the worker process starts is already possible using IIS Application Initialization module.

This is covered by https://github.com/aspnet/IISIntegration/pull/1402

If you wanted something else that isn't covered by this guidance, please feel free to open a new issue.

[sergioadh]

This issue has been opened several times and been moved/closed because of a lack of implementation. I've been following this issue since the beginning and nothing has really been done. Initialization is covered but always running is not, Kestrel goes down independent of application pool, it has been explained over and over again, and the issue just keeps getting closed/moved.

[shirhatti]

When used in conjunction with the new in-process hosting model, the Application Initialization module will achieve your desired behavior. Please try it out and give us your feedback.

[sergioadh]

So documentation for in-process hosting is scarce to say the least ... it hasn't even been mentioned in these issues that in-process hosting would be the way to go. A quick search gave me the following links:

https://github.com/aspnet/IISIntegration/issues/878 https://blogs.msdn.microsoft.com/webdev/2018/02/28/asp-net-core-2-1-0-preview1-improvements-to-iis-hosting/ https://berserkerdotnet.github.io/blog/aspnet-core-2-1-inprocess-hosting/

I will try and see if this issue is resolved by using in-process hosting, hopefully it is.

[jkotalik]

The feature was removed from 2.1 and punted to 2.2, which is still in preview. We are still working on documentation for the 2.2 release.

[orionstudt]

@jkotalik, for clarity's sake, do you mean that the in-process hosting feature mentioned in the 2.1 preview articles that @sergioadh linked was moved to 2.2?

It does actually sound like that would resolve the main issue, since then Kestrel would not be separate from the application pool.

[jkotalik]

Yes, it was moved to 2.2.

[Looooooka]

Moved to 2.2 but autostart doesn't work. Since there is an IISIntegration http request that happens when something goes wrong, instructing the netcore app to shutdown(guessing there's a route registered to handle this event... because clearly a http://host/iisintegration request can be logged) why not just add similar request for when an application pool is recycled or an application is stopped/started/restarted and then just raise the events already in place for these "scenarios": applicationLifetime.ApplicationStarted applicationLifetime.ApplicationStopped; applicationLifetime.ApplicationStopping This would honestly enable everyone to wire up their app from those points the way they needed to.

[motivks]

After setting up startMode=AlwaysRunning on the pool and preloadEnabled=true on the site, ASP.NET Core app be able to start automatically without any request.

https://blogs.msdn.microsoft.com/vijaysk/2012/10/11/iis-8-whats-new-website-settings/

[carlowahlstedt]

@jkotalik now that 2.2 is released, did the documentation get completed?

[Joelbear5]

How is this implemented when running Kestrel in Linux (without IIS as the reverse proxy server)?

We need to have control of loading (on start up instead of first request) and (prevention of) unloading at the application configuration and not externally controlled with IIS (ANCM).

[jkotalik]

@Looooooka with the release of 2.2, the module AspNetCoreModuleV2 should make startMode="AlwaysRunning" work. Please give it a try.

@carlowahlstedt we now have documentation on ANCM Inprocess in our docs. See https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/?view=aspnetcore-2.2#iis-options and https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-2.2

@Joelbear5 startMode="AlwaysRunning" is a feature of IIS and isn't implemented by Kestrel itself. If you are using a reverse proxy, there may be a feature to keep Kestrel alive.

[carlowahlstedt]

Thanks @jkotalik. What's the difference between the web.config hostingModel="InProcess" and the csproj <AspNetCoreHostingModel>InProcess</AspNetCoreHostingModel>? Are both needed?

[jkotalik]

@carlowahlstedt sorry for the incredibly late reply. InProcess is used to populate the web.config if there isn't one present in the project. If you have a web.config in your directory, it should always be used when you publish the application.

[YuliiaBabychKdd]

any updates? We have netcoreapp2.2, but still same issue:(

Any magic tricks or ideas how to fix? or when it will be fixed?

[binarypatrick]

We just wrote a tickle script

param($Uri)

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri $Uri | Select-Object -ExpandProperty StatusCode

[YuliiaBabychKdd]

@BinaryPatrick Thanks. Finished with same solution^^

[robbaman]

I've noticed that when you setup AlwaysRunning with a AspNet Core application (2.2) and I recycle the application (by clicking recycle in the IIS Manager) it correctly starts a new instance including the actual Kestrel-based application. This is also visible in the Windows Event Log with two events:

4-7-2019 12:04:24: Application 'D:\Sites\ShippingService\' started the coreclr in-process successfully. 4-7-2019 12:04:26: Application 'MACHINE/WEBROOT/APPHOST/SHIPPINGSERVICE' has shutdown.

Note how the Shutdown follows the start of a new instance. When the application pool lifecycle is managed/interrupted by an app_offline.html however, this does not work. Instead the logs are these:

4-7-2019 11:58:30: Application 'D:\Sites\ShippingService\' was recycled after detecting the app_offline file. 4-7-2019 11:58:33: Application 'MACHINE/WEBROOT/APPHOST/SHIPPINGSERVICE' has shutdown.

Notice a lack of the "started the coreclr in-process successfully" event in the logs. The Kestrel-application does indeed not start in this instance. This makes publishing a new version via Webdeploy.exe pretty hard. Once a single request comes in the site starts properly and everything works.

[unionthugface]

Sorry to reply to an old thread, but I am desperate. I have a .NET Core 2.2 app, in-process, hosted in IIS 10. I have application pool set to AlwaysRunning with Timeout=0, Preload=true; however my app still stops every 29 hours when the application pool recycles, and then does not restart itself until someone pings the URL. Back in the .NET Framework days I could write a bootstrapper class and hook into the Global.asax file/IIS ApplicationPreload settings; I haven't found any such thing for .NET Core and would so appreciate if someone could point me to one. My office is going nuts because we moved a recurring jobs handler app to .NET Core, and now it won't work because of this application pool recycle thing; every 29 hours on the dot, it stops. What am I missing? If I go back to my boss and tell him we need to write a tickle script like someone suggested, he'll be upset.