BackgroundService StopAsync not called when stopping Web App in Azure using ASP.NET Core 6 Minimal API

[simonsattinshell]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

I created the template Minimal API template with VS 2022 ASP.NET 6.0, and added a BackgroundService as a HostedService. I deployed this to Azure and it starts the Background service fine and i can see it in the logs.

However when i stop the web app in Azure, the StopAsync of the BackgroundService is not called. Do i need to hook something up in the Program.cs with the builder.Host? How in the code can i get notified that the web app is shutting down in case i need to do some other graceful shutdown? Program.cs

using MinAPI.Test.Workers;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

builder.Services.AddHostedService<Worker>();

var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();

app.UseHttpsRedirection();

var summaries = new[]
{
    "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};

app.MapGet("/weatherforecast", () =>
{
    var forecast = Enumerable.Range(1, 5).Select(index =>
       new WeatherForecast
       (
           DateTime.Now.AddDays(index),
           Random.Shared.Next(-20, 55),
           summaries[Random.Shared.Next(summaries.Length)]
       ))
        .ToArray();
    return forecast;
})
.WithName("GetWeatherForecast");

app.Run();

internal record WeatherForecast(DateTime Date, int TemperatureC, string? Summary)
{
    public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

Worker.cs

namespace MinAPI.Test.Workers
{
    public class Worker : BackgroundService
    {
        private readonly ILogger<Worker> _logger;

        public Worker(ILogger<Worker> logger)
        {
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
                await Task.Delay(1000, stoppingToken);
            }

            _logger.LogInformation("Worker cancellation token finished ");
        }

        public override Task StartAsync(CancellationToken cancellationToken)
        {
            _logger.LogWarning("Worker STARTING");
            return base.StartAsync(cancellationToken);
        }

        public override Task StopAsync(CancellationToken cancellationToken)
        {
            _logger.LogWarning("Worker STOPPING: {time}", DateTimeOffset.Now);
            return base.StopAsync(cancellationToken);
        }
    }
}

Expected Behavior

I would expect the line _logger.LogWarning("Worker STOPPING: {time}", DateTimeOffset.Now); to be hit when stopping in Azure.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

.Net 6

Anything else?

No response

[Tratcher]

Related: https://github.com/dotnet/aspnetcore/issues/22272

@bradygaster

[simonsattinshell]

Related: #22272

@bradygaster

This is a slightly different scenario. My post, I have setup a Web API project and added a worker service to that. The other post is a background service that gets deployed as a web job. On a different note, I've noticed that when a background service is deployed as webjob the logs act differently - for example if i log as a debug, it shows in the logstream that its logged as "Information dbug".

It's as if when deploying the Background worker service project as WebJob, its deployed to Azure as a web App Service that contains the worker service within the WebJob, and the web App service is capturing stdout logging from the WebJob.

When a background worker service is deployed to Azure, as WebJob, then i can work around graceful shutdown by watching for the webjob shutdown file; i asked a question on this on stackoverflow a while ago and manage to answer my own question https://stackoverflow.com/a/69986910/2126514.

However, when the project is a web project like this Web API one i have, and a worker is added to that, there doesn't appear to be a similar process of the webjob shutdown file (probably because this isn't a web job).

[ghost]

Thanks for contacting us. We're moving this issue to the .NET 7 Planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[bradygaster]

This is a known issue for which there is a similar existing issue specific to the Worker template prior to 6.0. App Service essentially sends a different shutdown command. There is no plan to resolve this in the app service side, nor known workaround when one is running their Worker in app service. Have you tried running your worker in Azure Container Apps or in Azure Container Instances? Those options don't have this same issue.

[bradygaster]

I see your situation is a little different, but I suspect will fall prey to the issue @Tratcher linked. So you have a web app I to which you also added a background worker class with a StopAsync, and you're not seeing the StopAsync fire when the app stops? As in, when you hit the "stop" button in the portal, for example?

[simonsattinshell]

Yes that is correct and exactly what i am doing: when i hit the stop button for the web app, the StopAsync is not called and the web app process is just killed almost immediately.

[rhyek]

Is there a recommended workaround? I've found that also implementing IAsyncDisposable works. So shutdown code would go in DisposeAsync and not StopAsync.

Edit: Actually, NVM. This is working as expected:

public class MonitorService : BackgroundService
{
    private readonly ILogger<MonitorService> _logger;

    public MonitorService(ILogger<MonitorService> logger)
    {
        _logger = logger;
        _logger.LogInformation("Constructed");
    }

    public override async Task StartAsync(CancellationToken cancellationToken)
    {
        await Task.Delay(millisecondsDelay: 2_000,
            cancellationToken: CancellationToken.None);
        _logger.LogInformation("StartAsync");

        await base.StartAsync(cancellationToken: cancellationToken);
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await Task.Delay(millisecondsDelay: 2_000,
            cancellationToken: CancellationToken.None);
        _logger.LogInformation("ExecuteAsync");
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        await base.StopAsync(cancellationToken: cancellationToken);

        await Task.Delay(millisecondsDelay: 2_000,
            cancellationToken: CancellationToken.None);
        _logger.LogInformation("StopAsync");
    }
}

output:

info: Core.Monitor.MonitorService[0]
      Constructed
info: Core.Monitor.MonitorService[0]
      StartAsync
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://0.0.0.0:3000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/carlos/Dev/personal/banco-industrial-monitor-dotnetcore/src/BancoIndustrialMonitor/HttpApi/
info: Core.Monitor.MonitorService[0]
      ExecuteAsync
^Cinfo: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
info: Core.Monitor.MonitorService[0]
      StopAsync

Process finished with exit code 0.

[tdw-hughes]

A possible additional piece of info - I have had the same problem (in ASP.NET Core 5.0), but have noticed this particular behaviour:

1. Deployed the site to IIS with preload and always running enabled.
2. Start AppPool - the hosted service starts. Stop AppPool - StopAsync is not called. i.e. problem is as described above.

However, if, after starting the AppPool, I make an HTTP request to the controller associated with this site, and then stop the AppPool after that, then StopAsync is actually called correctly.

[adeliab]

I have the same issue with .NET 6. I have a background service listening to a service bus using ServiceBusClient. On stop async I logs that the method is being called and then calls ServiceBusClient.DisposeAsync() I ran the worker service in VS with and without Docker Support. When i stop the application/ container, I don't see the log being displayed

[Euan-McVie]

Is there any follow up with at least a workaround for this?

Without the StopAsync I am not clear on how I can run cleanup code from my background service. This is a requirement as the service is interacting with a 3rd party that requests a clean logout flow as part of their certification to use their services.

Thanks

Note that the issue happens even running in console mode locally using Ctrl+c so does not appear to be linked to Azure specific deployments and looks more like a SDK bug with using hosted services inside a web app.

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

// Configure the AppConfiguration
builder.Host.ConfigurePowerDeskAppConfiguration();

// Add services to the container.
builder.Services.AddControllers();
builder.Services.AddHostedService<M7Service>();
builder.Services.AddEpexM7(builder.Configuration);
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddAdapterOptions(builder.Configuration);

WebApplication app = builder.Build();

// Configure the HTTP request pipeline.

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

[tdw-hughes]

Hi Euan, I can tell you how we worked around this:

in startup.cs:

Register some methods here:

        public void Configure(IApplicationBuilder app, IHostApplicationLifetime applicationLifetime, IWebHostEnvironment env, ILoggerFactory loggerFactory)
        {
            applicationLifetime.ApplicationStopping.Register(OnShutdown);
            applicationLifetime.ApplicationStarted.Register(OnStarted);

        ... etc. ...
        }

Add this code. We created a dummy PingController to use as an endpoint. It didn't seem to matter whether the Controller returned successful or we got an authorisation error - it worked to make ASP register properly and then the expected shutdown calls are made.

        private void OnStarted()
        {
            // Force a request on the controller, as there is a bug in ASP.NET that doesn't shutdown services gracefully at service stop until at least one request received
            Task.Factory.StartNew(() =>
            {
                try
                {
                    var client = new HttpClient();
                    client.BaseAddress = new Uri("whatever path is to this service");
                    var result = client.SendAsync(new HttpRequestMessage(HttpMethod.Get, "api/Ping" )).Result;
                    Console.WriteLine($"Pinged an API controller to ensure shutdown will register correctly: {result.StatusCode}");
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                }
            });
        }

        private void OnShutdown()
        {
            //noop - unless you have anything specific you need to do here.
        }

[Euan-McVie]

Many thanks Unfortunately that didn't work when running locally as a console app, so disrupts our dev workflows. Not sure why and haven't investigated further. I used the lifetime approach suggested, but embedded it inside the hosted service and this works, although requires any of the hosted services that need shutdown code to implement and while I don't particularly like the sync over async, as this is only for shutdown its not really a big issue.

    public M7Service(
        IHostApplicationLifetime lifetime,
        IM7ConnectionManager m7ConnectionManager,
        IM7RpcClient m7RpcClient,
        IOptions<ConnectionOptions> options,
        ILogger<M7Service> logger)
    {
        lifetime.ApplicationStopping.Register(OnStopping);
    }

    private void OnStopping() => DisconnectAsync(CancellationToken.None).GetAwaiter().GetResult();

[jez9999]

This is a known issue for which there is a similar existing issue specific to the Worker template prior to 6.0. App Service essentially sends a different shutdown command. There is no plan to resolve this in the app service side, nor known workaround when one is running their Worker in app service. Have you tried running your worker in Azure Container Apps or in Azure Container Instances? Those options don't have this same issue.

Out of interest why is it such a difficult fix for App Service just to send a SIGTERM? It would seem to be a one-liner.

[AKomyshan]

Still not fixed in .NET 8?

[SamandarbekYR]

Still not fixed in .NET 8?

Yes Not Fixed