BackgroundService stops abruptly when awaiting

[woutervanranst]

I LOVE the cleanness/compactness of the syntax, so I was creating a CLI which runs a backgroundservice. However, it terminates abruptly when it hits an actual await --

Consider this minimal example

var builder = CoconaApp.CreateBuilder();
builder.Services.AddHostedService<Service>();

var app = builder.Build();
app.AddCommand(() => { Console.WriteLine("Hello"); });
await app.RunAsync();

class Service : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (true)
        {
            //OPTION 1: Thread.Sleep(1000);
            //OPTION 2: await Task.Delay(1000);
        }
    }
}

with Option 1 uncommented, the app goes in an endless cycle (as expected) with Option2 uncommented, the app shuts down abruptly, no exceptions thrown.

When using Host.CreateDefaultBuilder, this behaves as expected

var builder = Host.CreateDefaultBuilder(args);
builder.ConfigureServices((hostContext, services) =>
    {
        services.AddHostedService<Service>();
    });

var app = builder.Build();
await app.RunAsync();

class Service : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (true)
        {
            await Task.Delay(100);
            //Thread.Sleep(1000);
        }
    }
}

[mayuki]

This is because Cocona terminates Host after executing the command. Host has a default timeout of 30 seconds waiting for IHostedService to complete. https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.hosting.hostoptions.shutdowntimeout?view=dotnet-plat-ext-8.0 https://github.com/dotnet/runtime/blob/bec45e34cbce7ff9378f05ab70b6fb147ebdabbc/src/libraries/Microsoft.Extensions.Hosting/src/HostOptions.cs#L24

builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromMinutes(10);
});

[woutervanranst]

Hm; this did not occur after 30 seconds and while the command execution was still in flight :/ it literally was while hitting the first await/continuation

[mayuki]

Apparently, Microsoft.Extensions.Hosting 6.0.0 had a timeout of 5 seconds. Cocona 2.2.0 depends on 6.0 as the minimum version. https://github.com/dotnet/runtime/blob/release/6.0/src/libraries/Microsoft.Extensions.Hosting/src/HostOptions.cs#L18

[woutervanranst]

So if I want to have a BackgroundService running in Cocona - how should I do it? An 'eternal' ShutdownTimeout?

[MrKacafirekCZ]

So if I want to have a BackgroundService running in Cocona - how should I do it? An 'eternal' ShutdownTimeout?

I don't have a perfect solution. Rather more of a hacky solution that works for the time being. It requires a little bit of knowledge of how ASP.NET applications work, so bear with me.

Since Cocona is trying to shutdown the application the moment a command finishes executing, in order to have a BackgroundService still running, you need to somehow prevent the application from shutting down.

Hacky solution 1 - Prevent the command from finishing.

Let's make it so the command never finishes!

internal static class Program
{
    public static void Main()
    {
        CoconaAppBuilder builder = CoconaApp.CreateBuilder();
        builder.Services.AddHostedService<MyService>();

        CoconaApp app = builder.Build();
        app.AddCommand(async () =>
        {
            Console.WriteLine("Hello from command!");

            await Task.Delay(Timeout.Infinite); // Prevents the command from finishing
        });
        app.Run();
    }
}

internal class MyService : BackgroundService
{
    protected async override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (true)
        {
            Console.WriteLine("Hello from service!");

            //Thread.Sleep(1000);
            await Task.Delay(1000);
        }
    }
}

The result:

$ dotnet run
Hello from service!
Hello from command!
Hello from service!
Hello from service!
Hello from service!
Hello from service!
Hello from service!
Hello from service!
...

Well that looks like it's fixed! Hooray! Now here's the real problem:

If you upgrade MyService into something like this:

internal static class Program
{
    ...
}

internal class MyService : BackgroundService
{
    private readonly IHostApplicationLifetime _lifetime;

    public MyService(IHostApplicationLifetime lifetime)
    {
        _lifetime = lifetime;
    }

    protected async override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        int i = 10;

        while (i > 0)
        {
            Console.WriteLine($"Hello from service! {i}");

            //Thread.Sleep(1000);
            await Task.Delay(1000);
            i--;
        }

        _lifetime.StopApplication();
    }
}

...and run the application now, the result is:

$ dotnet run
Hello from service! 10
Hello from command!
Hello from service! 9
Hello from service! 8
Hello from service! 7
Hello from service! 6
Hello from service! 5
Hello from service! 4
Hello from service! 3
Hello from service! 2
Hello from service! 1

$

If you're just reading this without trying the code for yourself, you will not see the problem. However if you are then you've probably noticed the odd delay after Hello from service! 1. That's the ShutdownTimeout from the class HostOptions kicking into action. If you set the ShutdownTimeout to a ridiculously high number like 10 minutes either by accident or intentionally using the method mentioned above:

builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromMinutes(10);
});

...you are going to be waiting 10 minutes for the application to properly shutdown.

In a small application, the chances of this happening are very unlikely but in a much bigger application with multiple commands, maybe even multiple background services and who knows what else, the chances are much higher.

You may have also noticed that you need to put await Task.Delay(Timeout.Infinite); in every single command that you create where you want your background service to be running.

This may not be a solution that works well for you so what about a different one?

Hacky solution 2 - Prevent the application from stopping.

Let's get back to the first simple example:

internal static class Program
{
    public static void Main()
    {
        CoconaAppBuilder builder = CoconaApp.CreateBuilder();
        builder.Services.AddHostedService<MyService>();

        CoconaApp app = builder.Build();
        app.AddCommand(() => { Console.WriteLine("Hello from command!"); });
        app.Run(); // I want you to focus on this line.
    }
}

internal class MyService : BackgroundService
{
    protected async override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (true)
        {
            Console.WriteLine("Hello from service!");

            //Thread.Sleep(1000);
            await Task.Delay(1000);
        }
    }
}

...and I want to you to focus on the line with app.Run();. And by that I mean how it works.

This right here is a rough example of how it works in the background:

internal static class Program
{
    public static void Main()
    {
        CoconaAppBuilder builder = CoconaApp.CreateBuilder();
        builder.Services.AddHostedService<MyService>();

        CoconaApp app = builder.Build();
        app.AddCommand(() => { Console.WriteLine("Hello from command!"); });
        //app.Run();

        IHost host = app.Services.GetRequiredService<IHost>();

        try
        {
            host.StartAsync().GetAwaiter().GetResult();

            host.WaitForShutdownAsync().GetAwaiter().GetResult();
        }
        finally
        {
            if (host is IAsyncDisposable asyncDisposable)
            {
                asyncDisposable.DisposeAsync().GetAwaiter().GetResult();
            }
            else
            {
                host.Dispose();
            }
        }
    }
}

internal class MyService : BackgroundService
{
    ...
}

If you run this code, you will see that nothing has changed. We still get the output Hello from service! from our MyService class, we still get the output Hello from command! and we still have the problem of the application abruptly shutting down after a little while.

I want you to focus on this line host.WaitForShutdownAsync().GetAwaiter().GetResult();. If you replace this line with something like this:

internal static class Program
{
    public static void Main()
    {
        CoconaAppBuilder builder = CoconaApp.CreateBuilder();
        builder.Services.AddHostedService<MyService>();

        CoconaApp app = builder.Build();
        app.AddCommand(() => { Console.WriteLine("Hello from command!"); });
        //app.Run();

        IHost host = app.Services.GetRequiredService<IHost>();

        try
        {
            host.StartAsync().GetAwaiter().GetResult();

            //host.WaitForShutdownAsync().GetAwaiter().GetResult();
            Thread.Sleep(Timeout.Infinite);
        }
        finally
        {
            if (host is IAsyncDisposable asyncDisposable)
            {
                asyncDisposable.DisposeAsync().GetAwaiter().GetResult();
            }
            else
            {
                host.Dispose();
            }
        }
    }
}

internal class MyService : BackgroundService
{
    ...
}

..and run the application now, the result is:

$ dotnet run
Hello from service!
Hello from command!
Hello from service!
Hello from service!
Hello from service!
Hello from service!
Hello from service!
Hello from service!
...

Hooray! It doesn't shutdown the application and you no longer have to worry about putting await Task.Delay(Timeout.Infinite); in every command. Now here is the real BIG problem:

Congratulations! You've just deleted any and all ways to gracefully shutdown the application no matter how hard you're gonna try.

Changing the ShutdownTimeout to 1 second?

builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromSeconds(1);
});

That will do nothing.

Spamming lifetime.StopApplication() everywhere?

IHostApplicationLifetime lifetime = _serviceProvider.GetRequiredService<IHostApplicationLifetime>();

lifetime.StopApplication();
lifetime.StopApplication();
lifetime.StopApplication();
lifetime.StopApplication();
lifetime.StopApplication();

Sorry, that won't do anything either.

What about Environment.Exit(0);? Not a graceful application shutdown.

The real solution?

So far the only theoretical solution I've come up with is to create a new attribute that would serve as a boolean flag whether or not the application should shutdown right after the command finishes executing.

Usage would be something like this:

app.AddCommand(() => { Console.WriteLine("Hello from command!"); }).WithMetadata(new DoNotShutdownAppAttribute());

...or this:

public class MyCommands
{
    [Command("helloworld")]
    [DoNotShutdownApp]
    public void HelloWorld()
    {
        Console.WriteLine("Hello world!");
    }
}

...and then somehow have a condition for this line (_lifetime.StopApplication();) in the package in the class CoconaHostedService: https://github.com/mayuki/Cocona/blob/master/src/Cocona/Hosting/CoconaHostedService.cs#L77

[woutervanranst]

Thanks @MrKacafirekCZ - that was certainly more than I was expecting as an answer; appreciate the explanation and I also understand why it behaves like it does now.

At this point I m thinking two things:

• what makes Cocona behave different vs the CreateDefaultBuilder - I thought (wo reviewing the source code) the one was built on the other and thus the behavior would be similar
• I d probably make a CoconaBackgroundService that exposes a TaskCompletionSource that is set when the backgroundservice is finished - and then an override in the RunAsync that scans the assembly for all CoconaBackgroundServices and does a Task.WaitAll for all these, but arguably this is a tricky setup for a developer to get right