WebJobs: Alternate Graceful Shutdown signal

[mathewc]

As documented here, for Continuous jobs, the job host signals graceful shutdown via the WEBJOBS_SHUTDOWN_FILE. The job code can listen for this file and perform any shutdown operations required before the host kills the process.

Currently it's up to the job code to handle listening for this file. The WebJob SDK handles this automatically (watcher code here), and initiates listener shutdown via cancellation token. I.e. it cancels the token that is flowed to all the various function listeners. When not using the WebJob SDK, the onus is on the user's job code to handle this.

However, in the case where the WebJobs SDK is handling the shutdown file notification, it appears that currently this shutdown doesn't cause IHostedService.StopAsync to be called, meaning any hosted service registered by the SDK or users are not stopped. So we have some work in the WebJobs SDK to do to ensure everything shuts down properly. Likely this means that WebJobsShutdownWatcher needs to use IHostLifetime to call StopAsync when triggered. Update: The WebJobs SDK issue has been addressed by https://github.com/Azure/azure-webjobs-sdk/pull/2730.

Some ASP.NET Core users are complaining that when deploying generic host jobs to AppService as a Continuous WebJob (example code below), shutdown isn't handled properly. Here's a recent discussion of this issue. They expect things to "just work" - when the job is shutting down, they expect a ctrl+c to be sent to stdin for the process, which will then trigger the built-in generic host shutdown behavior (e.g. calling StopAsync on hosted services, etc.).

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureServices((hostContext, services) =>
            {
               services.AddHostedService<Worker>();
            });
}

Proposal: The Kudu job host should send a ctrl+c to the process stdin to trigger shutdown. Open question on whether we should do this in addition to the existing file based notification, or whether it needs to be opt-in. We need to consider whether adding this behavior by default would be a breaking change for any existing jobs. We could also consider making this configurable, with the default remaining file based.

@suwatch What do you think?

[suwatch]

I think that is a good idea. To avoid regression, it will be opt in feature (eg. via appsettings) - we can enable by default for WebJob SDK scenario. @mathewc it would help speed up if you could write a sample C# app to illustrate sending CTRL-C to the another process.

[ahmelsayed]

As far as I know, you need to pinvoke GenerateConsoleCtrlEvent() to send a CTRL-C signal from a C# application to another process in Windows. This is an example of how to do it in Kudu's code https://github.com/projectkudu/kudu/blob/fdffce21d012d7691224bd3814d114f958f7e54f/Kudu.Services/Commands/PersistentCommandController.cs#L76-L81

However, I also know that it didn't used to work in the sandbox due to limitations of how Windows Console APIs work in general inside the sandbox. I don't know if that's still true or not.

[jez9999]

1) Does this apply to web apps too? I've found that my web app's IHostedService doesn't run StopAsync either.

2) Why on Earth should this be opt-in? Of course I expect it to "just work" when I shut down the app, why else are methods like StopAsync provided? Using a totally different graceful shutdown method seems crazy to me. I don't see what regressions would happen because you start handling shutdown properly; if something goes wrong with any code because of that, that code is broken.

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure: https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano Azure App Service