search

hangfire-postgres / Hangfire.PostgreSql

PostgreSql Storage Provider for Hangfire
Other
358 stars 132 forks source link

Attempting to retrieve FetchedJobs crashes due to duplicates #370

Open Jameak opened 3 months ago

Jameak commented 3 months ago

We're getting an exception when attempting to retrieve the list of fetched jobs via the Monitoring API.

This line of code: JobStorage.Current.GetMonitoringApi().FetchedJobs("default", 0, int.MaxValue)

Causes the following exception:

System.ArgumentException: An item with the same key has already been added. Key: 24014
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at Hangfire.PostgreSql.PostgreSqlMonitoringApi.FetchedJobs(IEnumerable`1 jobIds)
   at Hangfire.PostgreSql.PostgreSqlMonitoringApi.FetchedJobs(String queue, Int32 from, Int32 perPage)
   at Somewhere.In.My.Code

The issue seems to occur in the following situation. This is purely based on logs from our environments, I have not had the time to attempt to reproduce it locally:

  1. A hangfire server is currently processing jobs.
  2. The hangfire server disappears from the active server list (such as when the server gets restarted or the application crashes)
  3. The other hangfire servers (or the 'new' server if it was restarted) notices that the jobs from step 1 might have been aborted, because they're marked as being processed by a server that is not in the active server list.

If you attempt to get the list of fetched jobs after step 3 has occurred, it seems to (sometimes?) crash. This eventually resolves itself, presumably when the job gets successfully processed.

Versions: Hangfire.PostgreSql: 1.20.8 Hangfire: 1.8.7 .NET version 8.0.202