Hangfire SQL Storage with Aspire hosting SQL Server, unable to prepare schema

[murh-lego]

Hi, has anyone encountered similar?

I'm unable to prepare a schema on service startup. I'm running .NET Aspire with SQL Server, simple mssql hosted in docker container.

Local db is working a-ok.

Aspire will run clean-slate on each startup. Meaning some time will pass until SQL container is up and running. However, when he is and when my migrations run through.

The service I have hangfire on keeps throwing the below logs.

This is my connection string:

Server=127.0.0.1,1433;User ID=sa;Password=your!strongPassword;TrustServerCertificate=true;Database=my-db

This is my setup:

services.AddHangfire(hfCfg => hfCfg
            .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
            .UseColouredConsoleLogProvider()
            .UseSimpleAssemblyNameTypeSerializer()
            .UseRecommendedSerializerSettings()
            .UseSqlServerStorage(
                () => new SqlConnection(connectionString),
                new SqlServerStorageOptions
                {
                    PrepareSchemaIfNecessary = true,
                }
            )
        );

This is the log:

2024-05-16 05:37:23 [ERROR] (Hangfire.Processing.BackgroundExecution) Execution BackgroundServerProcess is still in the Failed state for 00:01:44.5396221 due to an exception, will be retried no more than in 00:00:15
Microsoft.Data.SqlClient.SqlException (0x80131904): Invalid object name 'HangFire.Server'.

[murh-lego]

Ok it seems that migration couldn't be applied. Retries were invoked resulting in final message:

(Hangfire.SqlServer.SqlServerObjectsInstaller) Was unable to perform the Hangfire schema migration due to an exception. Ignore this message unless you've just installed or upgraded Hangfire.

Can I somehow manually invoke migration or perhaps increase the retries?

[murh-lego]

Ended up writing my own runner which tries to perform installation and adding the jobs

IScheduledJobsStartupBootstrapper just bundels all those AddOrUpdate calls in one spot in some other assembly

public sealed class ScheduledJobsStartupRunner(
    IScheduledJobsStartupBootstrapper bootstrapper,
    SqlServerConnectionString connectionString,
    ILogger<ScheduledJobsStartupRunner> logger
) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                SqlServerObjectsInstaller.Install(new SqlConnection(connectionString));

                bootstrapper.AddScheduledJobs();

                logger.LogInformation("Startup scheduled jobs added successfully!");
                break;
            }
            catch (Exception e)
            {
                logger.LogDebug(e, "Something went wrong bootstrapping scheduled jobs");
                logger.LogWarning("Could not bootstrap scheduled jobs, retrying");
                await Task.WhenAny(Task.Delay(TimeSpan.FromSeconds(5), stoppingToken));
            }
        }
    }
}

[odinserj]

Ok it seems that migration couldn't be applied. Retries were invoked resulting in final message:

Hm, do you have any ideas what was the exception? It should be logged if logging template includes exceptions.

[murh-lego]

Ok it seems that migration couldn't be applied. Retries were invoked resulting in final message:

Hm, do you have any ideas what was the exception? It should be logged if logging template includes exceptions.

Yes, database wasn't available for some time and retries reached threshold. Eventually the db was up, but the service couldn't recover from it.

I've added resilient initializer to migrate db and add jobs that start from application startup life