HangfireIO / Hangfire

An easy way to perform background job processing in .NET and .NET Core applications. No Windows Service or separate process required
https://www.hangfire.io
Other
9.44k stars 1.71k forks source link

Time synchronization issue in 1.8.6? #2330

Closed twojnarowski closed 11 months ago

twojnarowski commented 11 months ago

I have updated my Hangfire nugets (Hangfire, Hangfire.Core, Hangfire.AspNetCore, Hangfire.SqlServer) from 1.8.5 to 1.8.6 and I have a problem. Previously the jobs were running fine. Right now my jobs have this message: "Looks like the job was aborted – it is processed by server id which reported its heartbeat more than 1 minute ago. It will be retried automatically after invisibility timeout, but you can also re-queue or delete it manually." The jobs completed show with an information that they were completed... 2 minutes in the future from now: image

I haven't changed anything about the configuration of Hangfire, didn't touch any system clocks on the app server or sql server where Hangfire stores my jobs.

This is my configuration:

builder.Services.AddHangfire(
                config => config
                    .UseSimpleAssemblyNameTypeSerializer()
                    .UseRecommendedSerializerSettings()
                    .UseSqlServerStorage(builder.Configuration.GetConnectionString("HangFire"))
                    .WithJobExpirationTimeout(TimeSpan.FromDays(90)));

            builder.Services.AddHangfireServer();

Here is the issue visible in one picture: image The heartbeat was properly recorded a few seconds ago, but the time at the bottom bar is 3 minutes into the future.

twojnarowski commented 11 months ago

It would appear so that our application server had the clock set to 2 minutes too fast, and synchronizing it with a time server to correct time made the Hangfire problem dissapear, but it had this clock set since the beginning, so the new version changed something that made this problem show up.

odinserj commented 11 months ago

Since 1.8.0 storage now can act as a time authority, but Dashboard UI doesn't use it yet for these pages, hence the issue. Thanks for reporting this, I will add corresponding logic in the upcoming 1.8.7 to make it more obvious.