HangfireIO / Hangfire

An easy way to perform background job processing in .NET and .NET Core applications. No Windows Service or separate process required
https://www.hangfire.io
Other
9.43k stars 1.7k forks source link

Deadlock found when trying to get lock; try restarting transaction #2470

Open srad opened 2 days ago

srad commented 2 days ago

For a while now I get each time I start my server which is running Hangfire immediately following error message:

2024-11-19 09:01:45.0532|0|ERROR|Hangfire.Processing.BackgroundExecution|Execution BackgroundServerProcess is in the Failed state now due to an exception, execution will be retried in no more than 00:00:15 NHibernate.Exceptions.GenericADOException: could not insert: 

[Hangfire.FluentNHibernateStorage.Entities._Server#workstation:8136:836b1a05-468e-48a8-a7de-2beeeec226ac][SQL: INSERT INTO `Hangfire_Server` (`Data`, `LastHeartbeat`, `Id`) VALUES (?, ?, ?)]

 ---> MySql.Data.MySqlClient.MySqlException (0x80004005): Deadlock found when trying to get lock; try restarting transaction  at MySql.Data.MySqlClient.MySqlStream.ReadPacketAsync(Boolean execAsync)
...

This is .net 8 with Hangfire 1.8.15 with MySQL Server 8.4.3-1.el9 and following config:

builder.Services.AddHangfire(configuration => configuration
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseStorage(FluentNHibernateStorageFactory.For(ProviderTypeEnum.MySQL,
        builder.Configuration.GetConnectionString("HangfireConnection"), new FluentNHibernateStorageOptions
        {
            DistributedLockWaitTimeout = TimeSpan.FromMinutes(30),
            TransactionIsolationLevel = IsolationLevel.Serializable,
            QueuePollInterval = TimeSpan.FromSeconds(15),
            JobExpirationCheckInterval = TimeSpan.FromHours(1),
            CountersAggregateInterval = TimeSpan.FromMinutes(5),
            UpdateSchema = true,
            DashboardJobListLimit = 1000,
            InvisibilityTimeout = TimeSpan.FromHours(12),
            TransactionTimeout = TimeSpan.FromHours(12),
            DefaultSchema = null, // use database provider's default schema,
            TablePrefix = "Hangfire_" // underscore necessary
        })));

There are no open jobs in the queue and I think everything is running fine but I still get every single time this exception.

Anyone knows what's going on here with the lock?