Shutting down BackgroundJobServer

[altyne]

When stopping the BackgroundJobServer it creates a logs: "Stopping the server due to DomainUnload or ProcessExit event...

Server hostname:10156:399b85ad caught shutdown signal...

Server hostname:10156:399b85ad stopped non-gracefully due to ServerWatchdog, ServerJobCancellationWatcher, MongoExpirationManager, MongoNotificationObserver, Worker, DelayedJobScheduler, RecurringJobScheduler. Outstanding work on those dispatchers could be aborted, and there can be delays in background processing. This server instance will be incorrectly shown as active for a while. To avoid non-graceful shutdowns, investigate what prevents from stopping gracefully and add CancellationToken support for those methods."

I tried calling

• Stop()
  • SendStop()
  • Dispose()
• WaitForShutdown*

Why the job server does not stop all instances even with infinite timeout option.

I know this is minimal but for logs pov looks the application was closed forcefully. Is this normal for application to have this logs?

[AdamCaviness]

I too get "Stopping the server due to DomainUnload or ProcessExit event..." when stopping my Windows Service (BackgroundService) hosted ASP.NET Core web app. Since I configure via services.AddHangfire as found here, I don't know how I'm supposed to gracefully shutdown Hangfire. In my case, this occurs with no jobs running.

Update

In my case, I was able to call the IHost's WaitForShutdownAsync() and it cleans up successfully now.

[odinserj]

@AdamCaviness could you show us what code you've changed? Also you can configure BackgroundJobServerOptions.StopTimeout to give background jobs some time to finish themselves without triggering their cancellation tokens immediately on shutdown. In this case no new jobs will be processed after shutdown is initiated, but current jobs will be allowed to finish.

[AdamCaviness]

@odinserj In my startup class, I configure Hangfire like this (none of this changed):

// Add Hangfire.
services.AddHangfire(config =>
{
    config.UseSimpleAssemblyNameTypeSerializer();
    config.UseActivator(new HangfireJobActivator());
    config.UseLogProvider(new HangfireLogProvider());
    config.UseSerializerSettings(new HangfireSerializerSettings());
    config.SetDataCompatibilityLevel(CompatibilityLevel.Version_170);
    config.UseSqlServerStorage(_jobsConnectionString, new SqlServerStorageOptions
    {
        CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
        SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
        QueuePollInterval = TimeSpan.Zero,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true
    });
});
services.AddHangfireServer();

My netcore BackgroundService invokes my own class that builds the IHostBuilder and then runs the IHost. I named this class WebServer. What I was missing in my WebServer class was a way to stop the host. So, I added my own StopAsync() method to WebServer, taking the BackgroundService's CancellationToken from it's overridden StopAsync method and passing it to the IHost's WaitForShutdownAsync.

public async Task StopAsync(CancellationToken cancellationToken)
{
    await _host?.WaitForShutdownAsync(cancellationToken);
}

I no longer get "Stopping the server due to DomainUnload or ProcessExit event..." in my log file when shutting down the Windows Service from the Service Control Manager. To be clear, running via F5 in Visual Studio and then stopping will not run through the shutdown code. Running via F5 also never outputs the original DomainUnload warning like it does when stopping the service via the Service Control Manager.

[odinserj]

Ah thanks, so there was an external bootstrapping implementation. I've added those app domain unload / process exit hooks to be able to do at least anything when process is stopped unexpectedly without any waiting. Hangfire is able to retry anything even without those hooks, but they try to minimise the delays between failure occurrence and failure detection.

[RunnerM]

@AdamCaviness Can you maybe elaborate a bit more on how your bootstrapping implementation works? I get the concept but I don't know how to and what to wrap into the WebServer class of yours. Thanks