GetJobsWithProperties duplicate key exception

[AndreSteenbergen]

I regularly get the following exception in the retries section in hangfire. I don't know the root cause... Jobs might be double inserted into hangfire.

System.ArgumentException: An item with the same key has already been added. Key: c6b5dd4b-bf88-43b5-af05-cc1f8eb8fba8 at System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(Object key) at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add) at Hangfire.Redis.RedisMonitoringApi.GetJobsWithProperties[T](IDatabase redis, String[] jobIds, String[] properties, String[] stateProperties, Func4 selector) at Hangfire.Redis.RedisMonitoringApi.<Queues>b__16_0(IDatabase redis) at ClaxeCrawler.CronJobsProcessor.WebsiteQueueIsEmpty(String queueName) at ClaxeCrawler.CronJobsProcessor.<StartCrawlAction>d__13.MoveNext()

[AndreSteenbergen]

I hacked in a small work-around, in a private fork. But this doesn't take away the root cause. It's the jobIds = jobIds.Distinct().ToArray(); part

 private JobList<T> GetJobsWithProperties<T>(
            IDatabase redis,
            string[] jobIds,
            string[] properties,
            string[] stateProperties,
            Func<Job, List<string>, List<string>, T> selector)
        {
            if (jobIds.Length == 0) return new JobList<T>(new List<KeyValuePair<string, T>>());

            jobIds = jobIds.Distinct().ToArray();
            var jobs = new Dictionary<string, Task<RedisValue[]>>(jobIds.Length, StringComparer.OrdinalIgnoreCase);
            var states = new Dictionary<string, Task<RedisValue[]>>(jobIds.Length, StringComparer.OrdinalIgnoreCase);

            properties = properties ?? new string[0];

            var pipeline = redis.CreateBatch();

[MustafaJamal]

We faced same problem during beta testing of our application. Although after restarting IIS it went away. BUT we are suspicious about delivering out release due to this problem as it is show stopper. Can anyone please let me know when it will be properly fixed?

[IvanJosipovic]

I got this error today, I re-queued a job and it occurred. I had to delete my redis data for the problem to go away.

[marcoCasamento]

I've never had that error, if it happens again can you check the enqueued jobs directly querying Redis (prefix:queue:queueName] ?

[marcoCasamento]

Neither me or my colleagues ever face this error and yet we run a handful of servers that runs hundreds of thousands job a day. I suppose it's somehow configuration specific, I'm going to close this