Jobs are picked up from queues that the server is not supposed to listen to

[webseb]

I found several issues that sound similar but never a real solution.

I have two Hangfire servers. Let's call them A and B. Server A is configured to listen to QueueA (and only QueueA, no "default" queue!) and Server B listens to QueueB (also no "default" queue). I have a job that is scheduled on QueueA (doesn't matter if I schedule it as a background or a recurring job). But the job is (sometimes) picked up by Server B.

services.AddHangfireServer((sp, options) =>
{
    options.Queues = new[] { "queue_a" };
    options.WorkerCount = 1;
    options.ServerName = "server-a";
});

services.AddHangfireServer((sp, options) =>
{
    options.Queues = new[] { "queue_b" };
    options.WorkerCount = 1;
    options.ServerName = "server-b";
});

recurringJobManager.AddOrUpdate<JobA>(
                    nameof(JobA),
                    "queue_a",
                    job => job.Execute(CancellationToken.None),
                    "<some cron expression>");

The problem: Server B does not reference the same set of assemblies and is thus not even able to load the job from the database. Instead of letting the other server pick it up, Server B keeps trying until the retry attempts are exhausted and the job fails.

Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
 ---> System.IO.FileNotFoundException: Could not resolve assembly 'JobA'.
   at System.Reflection.TypeNameParser.ResolveAssembly(String assemblyName)
   at System.Reflection.TypeNameParser.GetType(String typeName, ReadOnlySpan`1 nestedTypeNames, String assemblyNameIfAny)
   at System.Reflection.TypeNameParser.NamespaceTypeName.ResolveType(TypeNameParser& parser, String containingAssemblyIfAny)
   at System.Reflection.TypeNameParser.Parse()
   at System.Reflection.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Assembly requestingAssembly, Boolean throwOnError, Boolean ignoreCase, Boolean extensibleParser)
   at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
   at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName) in C:\projects\hangfire-525\src\Hangfire.Core\Common\TypeHelper.cs:line 78
   at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 96
   --- End of inner exception stack trace ---
   at Hangfire.Storage.InvocationData.DeserializeJob() in C:\projects\hangfire-525\src\Hangfire.Core\Storage\InvocationData.cs:line 120
   at Hangfire.RecurringJobExtensions.TriggerRecurringJob(IBackgroundJobFactory factory, JobStorage storage, IStorageConnection connection, IProfiler profiler, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\RecurringJobExtensions.cs:line 115
   at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now) in C:\projects\hangfire-525\src\Hangfire.Core\Server\RecurringJobScheduler.cs:line 333

I have a job filter that makes sure the original queue is used, even on retries. So the issue is not related to the job being re-queued with the wrong queue.

Why does a server pick up jobs from queues, that it is not meant to listen to and how can i fix that?

BTW: This issue is not related to the storage used. I see it with SqlServer, Redis and PostgreSql.

[braiinzz]

I got the same issue. @webseb Did you figure out a solution for this problem?

[KieranFleckney]

I have the same issue too, can't seem to find any solution too this.

[webseb]

The solution is two-fold: 1) update to the latest Hangfire version. There is 1st class support for queues in the recent versions (i.e. overloads that take the queue name as a parameter) and older solutions like enqueuing jobs with new EnqueuedState("myQueue") are no longer necessary. 2) for recurring jobs switch to Hangfire.DynamicJobs

A more detailed description can be found on StackOverflow.

PS: If your jobs fail they will be re-queued in the "default" queue. That is by design. If you want to nail the jobs to a specific queue, you have to use the QueueAttribute on your jobs or write sth. similar, if you require more flexible logic.

[KieranFleckney]

@webseb I am on the latest version and using QueueAttributes and it still has a chance to be retried in a different queue.

[webseb]

@KieranFleckney are we talking about recurring jobs or regular background jobs? can you show some code?

[KieranFleckney]

@webseb I am talking about any job which fails then is auto retries.

Lets say you have, 2 servers, each processing a different queue. Lets say queue-a and queue-b. If a job which was scheduled for queue-a (which on the first run will be processed on the correct server) fails for one reason or another. There is a chance it will be the rety will be processed by server which is processsing queue-b. Find the code here: https://github.com/KieranFleckney/hangfire-bug

[webseb]

@KieranFleckney try one of the overloads of the Enqueue(...) method that takes the queue name as a parameter. Same goes for the AddOrUpdate(...) method. In addition I recommend that you use Hangfire.DynamicJobs to schedule your recurring jobs. That addon is specifically designed to prevent the wrong server picking up a recurring job.

[KieranFleckney]

@webseb Thank you for the suggestions.

Now passing the queue in with the Enqueue method does fix the problem if you are schudling one of tasks. Now doing it with AddOrUpdate method doesn't because a different server may try and schudele the recurring task and fail to load the assembly.

Which is what I presume the DynasmicJobs package is trying to fix. However, there seems to be a big issue with queues in that package (unfortantanly issues are not turned on for that package, so I can't raised a bug). When you set a queue using the AddOrUpdateDynamic method it works as normally. However, jobs are scheduled in the default queue even if you set it to a different queue. But it does work if you go into the dashboard and manaully enqueue the job then it will run in the correct queue. So it was soo close to fixing my problem but untill that bug is fixed this will continue to be a problem.

[webseb]

@KieranFleckney Try to pass a QueueAttribute when you call AddOrUpdateDynamic docs That's what worked for me.

[KieranFleckney]

@webseb thank you, I thought I tested that already but maybe I didn't anyway all working now.

[doanhieu9797]

But anyone have solution for this issue, i have same problem.

[webseb]

@doanhieu9797 Check out the solution on StackOverflow