Fix System.ArgumentException: An item with the same key has already been added. Key: LastExecution

[hquangthinh]

Hi @imranmomin appreciate it if you could spend sometimes to look at my PR for this issue https://github.com/imranmomin/Hangfire.AzureCosmosDb/issues/21

[imranmomin]

@hquangthinh - thank you for your time to investigate the issue and also for the fix. I will put some time during the weekend to accept and release the package

[tabishfriend]

Waiting for this fix. I am getting this exception

{"status":500,"code":"ERR_INTERNAL_SERVER_ERROR","message":"Internal server error","failure":"System.Collections.Generic.KeyNotFoundException: The given key 'EnqueuedAt' was not present in the dictionary.\r\n at TValue System.Collections.Generic.Dictionary<TKey, TValue>.get_Item(TKey key)\r\n at JobList Hangfire.Azure.CosmosDbMonitoringApi.EnqueuedJobs(string queue, int from, int perPage)+(State state, Job job, DateTime? fetchedAt) => { }\r\n at JobList Hangfire.Azure.CosmosDbMonitoringApi.GetJobsOnQueue(string queryText, string queue, Func<State, Job, DateTime?, T> selector)+(Queue queueItem) => { }\r\n at void System.Collections.Generic.List.ForEach(Action action)\r\n at JobList Hangfire.Azure.CosmosDbMonitoringApi.GetJobsOnQueue(string queryText, string queue, Func<State, Job, DateTime?, T> selector)\r\n at JobList Hangfire.Azure.CosmosDbMonitoringApi.EnqueuedJobs(string queue, int from, int perPage)\r\n at IList Hangfire.Azure.CosmosDbMonitoringApi.Queues()\r\n at void Hangfire.Dashboard.Pages.QueuesPage.Execute()\r\n at string Hangfire.Dashboard.RazorPage.TransformText(string body)\r\n at string Hangfire.Dashboard.RazorPage.ToString()\r\n at Task Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context)\r\n at async Task Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)\r\n at async Task Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n at async Task Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\r\n at async Task Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\r\n at async Task Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\r\n at async Task Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\r\n at async Task Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)+Awaited(?)","evidence":{"scheme":"http","location":"//jobs/jobs/enqueued","method":"GET","queryString":{"value":"","hasValue":false},"headers":{"connection":["keep-alive"],"accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9"],"accept-Encoding":["gzip, deflate"],"accept-Language":["en-US,en;q=0.9,da;q=0.8"],"host":["osi4263:12118"],"referer":["http://osi4263:12118/jobs/servers"],"user-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"],"upgrade-Insecure-Requests":["1"]},"body":"","coockies":[],"connection":{"id":"0HMAQVLU31UH3","localLocation":"::ffff:10.73.221.102:12118","remoteLocation":"::ffff:10.62.11.229:55855"}},"helpLink":"/api/help/e=ERR-500"}

[dioptre]

This might be unrelated, but might also fix:

FAIL|1643677788979|BackgroundExecution|9e652624df2a4d06860d579e476d112d|0|0|0|Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:16
XMSG|9e652624df2a4d06860d579e476d112d|System.ArgumentException: An item with the same key has already been added. Key: Queue
XMSG|9e652624df2a4d06860d579e476d112d|   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
XMSG|9e652624df2a4d06860d579e476d112d|   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
XMSG|9e652624df2a4d06860d579e476d112d|   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
XMSG|9e652624df2a4d06860d579e476d112d|   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Azure.CosmosDbConnection.GetAllEntriesFromHash(String key)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.RecurringJobExtensions.GetRecurringJob(IStorageConnection connection, String recurringJobId, ITimeZoneResolver timeZoneResolver, DateTime now)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Server.RecurringJobScheduler.EnqueueNextRecurringJobs(BackgroundProcessContext context)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext c
ontext)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
XMSG|9e652624df2a4d06860d579e476d112d|   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)

[dioptre]

@imranmomin did you review this PR?

[imranmomin]

I was able to review. The only issue is the order of the document return and it may not always be at the top. But anyway give it a try if that helps

[dioptre]

Thanks!

@hquangthinh does the document return order look ok to you?

[hquangthinh]

Thanks!

@hquangthinh does the document return order look ok to you?

We moved to Hangfire redis as the fix still cannot be guaranteed in multiple servers environment