search

marcoCasamento / Hangfire.Redis.StackExchange

HangFire Redis storage based on original (and now unsupported) Hangfire.Redis but using lovely StackExchange.Redis client
Other
452 stars 108 forks source link

InvalidOperationException: Operations that change non-concurrent collections must have exclusive access #91

Closed VoidMonk closed 3 years ago

VoidMonk commented 4 years ago

Hi,

We're using Hangfire 1.7.7 and Hangfire.Redis.StackExchange 1.8.0, in a .NET Core 3.1 Worker Service project, running on Linux.

We've integrated a slightly modified DisableMultipleQueuedItemsFilter filter for throttling jobs' execution.

This filter uses distributed locks on Redis, that's causing an exception in Hangfire.Redis.StackExchange, similar to some other InvalidOperationException issues reported earlier in this repo.

Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
 ---> System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
   at Hangfire.Redis.RedisLock..ctor(IDatabase redis, RedisKey key, Boolean holdsLock, TimeSpan holdDuration)
   at Hangfire.Redis.RedisLock.Acquire(IDatabase redis, RedisKey key, TimeSpan timeOut, TimeSpan holdDuration)
   at Hangfire.Redis.RedisLock.Acquire(IDatabase redis, RedisKey key, TimeSpan timeOut)
   at Hangfire.Redis.RedisConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at ClusterWorker.HangfireFilters.DisableMultipleQueuedItemsFilter.AddFingerprintIfNotExists(IStorageConnection connection, Job job) in /home/runner/work/ClusterWorker/HangfireFilters.cs:line 49
   at ClusterWorker.HangfireFilters.DisableMultipleQueuedItemsFilter.OnCreating(CreatingContext context) in /home/runner/work/ClusterWorker/HangfireFilters.cs:line 26
   at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
   at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   --- End of inner exception stack trace ---
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   at ClusterWorker.HangfireJobs.RefreshApiData(IJobCancellationToken CancellationToken) in /home/runner/work/ClusterWorker/HangfireJobs.cs:line 38

I'll appreciate if you can also look into the root cause of this issue, and a possible fix.

Thank you.

marcoCasamento commented 3 years ago

clean up old issues, Iassume the mentioned PR fixed it