search

imranmomin / Hangfire.AzureDocumentDB

Azure DocumentDB storage provider for Hangfire
https://www.hangfire.io/
MIT License
14 stars 10 forks source link

Execution RecurringJobScheduler is in the Failed state now due to an exception #61

Open mudasar opened 3 years ago

mudasar commented 3 years ago

Hello there, I am facing this issue when running multiple servers with the same database. there are some recurring jobs which are stuck to a single server that was turned off. Is there a direct way to trigger these jobs manually? Also, any solution to the following issue?

Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:16 System.ArgumentException: An item with the same key has already been added. Key: LastExecution

mudasar commented 3 years ago

complete error stack is here [15:05:48 ERR] HTTP GET /hangfire/recurring responded 500 in 1451.9199 ms System.ArgumentException: An item with the same key has already been added. Key: LastExecution at System.Collections.Generic.Dictionary2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary2.Add(TKey key, TValue value) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector, IEqualityComparer1 comparer) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector) at Hangfire.Azure.CosmosDbConnection.GetAllEntriesFromHash(String key) at Hangfire.Storage.StorageConnectionExtensions.GetRecurringJobDtos(IStorageConnection connection, IEnumerable1 ids) at Hangfire.Storage.StorageConnectionExtensions.GetRecurringJobs(JobStorageConnection connection, Int32 startingFrom, Int32 endingAt) at Hangfire.Dashboard.Pages.RecurringJobsPage.Execute() at Hangfire.Dashboard.RazorPage.TransformText(String body) at Hangfire.Dashboard.RazorPage.ToString() at Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context) at Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)

imranmomin commented 3 years ago

An item with the same key has already been added. Key: LastExecution

What you can do is find the duplicate Hash entry for this recurring job and delete the older document.

select * from c where type = 6 and key = 'recurring-job-id'

mudasar commented 3 years ago

Hi, @imranmomin thanks for your reply. I did fix it by going through the cosmos DB for recurring jobs that were a lot. This happens when you try to join an already running job queue server.

mudasar commented 3 years ago

Is there a way we can avoid getting into this position at all ?

dioptre commented 2 years ago

This broke for us too.