Recurring job '?' can not be scheduled due to job load exception.

[gandarez]

Talking about Scalability/Distrubted Systems I think each server might see only its queues. In my production envrionment I'm running two instances in two servers. Each server actually runs over different queues as below

Server "A" has only libraries needed there and in the other server ("B") only the libraries needed also. When server "B" tries to Schedule a job it throws JobLoadException because there's no library into its binary folder. What I suggest is to change few classes to restrict RecurringJobScheduler class to watch only the right queues.

The queue is already in the context and could be used to filter recurring-jobs as well.

[jmvermeulen]

Having the exact same issue. To me this is clearly a bug, can anyone confirm?

[RupertAvery]

Also encountered this. Although it is merely a warning, it is unnecessarily cluttering the logs.

Also, can the Queue attribute be used for recurring jobs as well?

[LucasFarley]

There is solution for this problem? I am facing the same problem.

[ZlobnyiSerg]

+1 for this

[berdem]

+111

[GeXiaoguo]

Here is a solution.

[Deantwo]

Here is a solution.

I would call it a workaround, not a solution. The bug needs to be fixed in Hangfire and not ignored for another 4 years.

But thanks for sharing.

[GeXiaoguo]

Here is a solution.

I would call it a workaround, not a solution. The bug needs to be fixed in Hangfire and not ignored for another 4 years.

But thanks for sharing.

Yes, it is a workaround. This problem has been reported by many people for many years now. Nothing has happened yet. I am losing hope.

[udlose]

any update on this (over 4 years now)!

[christopher-bonitz]

No fixes yet, I just hit this issue today running on MongoDB. Now I have to run a seperate hangfire application (dashboard, server and database) just to run my "report" queue. @odinserj any chance you can fix this with your magic fingers?

[odinserj]

If this problem appears when using the same job storage for multiple applications with different code base, the simplest solution would be to use some shared assembly referenced from those multiple applications and put background job class and method (or at least their interfaces) there.

Recurring job schedule itself doesn't depend on the queue name, and when wrong server tries to create and schedule such a recurring job, it fails. And it is important to have access to the declared method, because filters in Hangfire can significantly change the background processing workflow.

I'm aware of this problem and really sorry there were a lot of troubles with it. I was trying to solve it many times and looks like it will be finally possible to do this after 1.8.0 is released with some tricks and not so hard limitations – by creating a shared assembly that will accept everything including filters dynamically and releasing this as an extension.

But currently the only workaround is to use some shared assembly referenced by all of the projects.

[christopher-bonitz]

If this problem appears when using the same job storage for multiple applications with different code base, the simplest ... But currently the only workaround is to use some shared assembly referenced by all of the projects.

You are honestly a fantastic and helpful person. I have begun the code migration to a shared space, crossing fingers for a fix in 1.8.0.

[afonsoft]

My suggestion is to create a JobFilter to do the Queue and Retry treatment together. The problem is in the ScheduledState class that does not use the Queue, while the EnqueuedState class has the Queue. I created an Attribute with the name AutomaticRetryInQueueAttribute, but it does not have the time waiting to boot the next job.

Don't have delay between in retry.

` public class AutomaticRetryInQueueAttribute : JobFilterAttribute, IApplyStateFilter, IElectStateFilter { private string queue; private int attempts; private readonly object _lockObject = new object();

    private readonly ILog _logger = LogProvider.For<AutomaticRetryInQueueAttribute>();

    public AutomaticRetryInQueueAttribute()
    {
        queue = "Default";
        attempts = 10;
    }

    public int Attempts
    {
        get { lock (_lockObject) { return attempts; } }
        set
        {
            if (value < 0)
            {
                throw new ArgumentOutOfRangeException(nameof(value), @"Attempts value must be equal or greater than zero.");
            }

            lock (_lockObject)
            {
                attempts = value;
            }
        }
    }

    public string Queue
    {
        get { lock (_lockObject) { return queue; } }
        set
        {
            lock (_lockObject)
            {
                queue = value;
            }
        }
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        var newState = context.NewState as EnqueuedState;
        if (!string.IsNullOrWhiteSpace(queue) && newState != null && newState.Queue != Queue)
        {
            newState.Queue = String.Format(Queue, context.BackgroundJob.Job.Args.ToArray());
        }

        if (context.NewState is ScheduledState &&
            context.NewState.Reason != null &&
            context.NewState.Reason.StartsWith("Retry attempt"))
        {
            transaction.AddToSet("retries", context.BackgroundJob.Id);
        }
    }

    public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        if (context.OldStateName == ScheduledState.StateName)
        {
            transaction.RemoveFromSet("retries", context.BackgroundJob.Id);
        }
    }

    public void OnStateElection(ElectStateContext context)
    {
        var failedState = context.CandidateState as FailedState;
        if (failedState == null)
        {
            // This filter accepts only failed job state.
            return;
        }

        var retryAttempt = context.GetJobParameter<int>("RetryCount") + 1;

        if (retryAttempt <= Attempts)
        {
            ScheduleAgainLater(context, retryAttempt, failedState);
        }
        else
        {
            _logger.ErrorException($"Failed to process the job '{context.BackgroundJob.Id}': an exception occurred.", failedState.Exception);
        }
    }

    private void ScheduleAgainLater(ElectStateContext context, int retryAttempt, FailedState failedState)
    {
        context.SetJobParameter("RetryCount", retryAttempt);

        const int maxMessageLength = 50;
        var exceptionMessage = failedState.Exception.Message.Length > maxMessageLength
            ? failedState.Exception.Message.Substring(0, maxMessageLength - 1) + "…"
            : failedState.Exception.Message;

        // If attempt number is less than max attempts, we should
        // schedule the job to run again later.

        var reason = $"Retry attempt {retryAttempt} of {Attempts}: {exceptionMessage}";

        context.CandidateState = (IState)new EnqueuedState
        {
            Reason = reason,
            Queue = String.Format(Queue, context.BackgroundJob.Job.Args.ToArray())
        };

        _logger.WarnException($"Failed to process the job '{context.BackgroundJob.Id}': an exception occurred. Retry attempt {retryAttempt} of {Attempts} will be performed.", failedState.Exception);
    }
}

`

[christopher-bonitz]

My suggestion is to create a JobFilter to do the Queue and Retry treatment together. The problem is in the ScheduledState class that does not use the Queue, while the EnqueuedState class has the Queue. I created an Attribute with the name AutomaticRetryInQueueAttribute, but it does not have the time waiting to boot the next job.

Don't have delay between in retry.

` public class AutomaticRetryInQueueAttribute : JobFilterAttribute, IApplyStateFilter, IElectStateFilter { private string queue; private int attempts; private readonly object _lockObject = new object();

    private readonly ILog _logger = LogProvider.For<AutomaticRetryInQueueAttribute>();

    public AutomaticRetryInQueueAttribute()
    {
        queue = "Default";
        attempts = 10;
    }

    public int Attempts
    {
        get { lock (_lockObject) { return attempts; } }
        set
        {
            if (value < 0)
            {
                throw new ArgumentOutOfRangeException(nameof(value), @"Attempts value must be equal or greater than zero.");
            }

            lock (_lockObject)
            {
                attempts = value;
            }
        }
    }

    public string Queue
    {
        get { lock (_lockObject) { return queue; } }
        set
        {
            lock (_lockObject)
            {
                queue = value;
            }
        }
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        var newState = context.NewState as EnqueuedState;
        if (!string.IsNullOrWhiteSpace(queue) && newState != null && newState.Queue != Queue)
        {
            newState.Queue = String.Format(Queue, context.BackgroundJob.Job.Args.ToArray());
        }

        if (context.NewState is ScheduledState &&
            context.NewState.Reason != null &&
            context.NewState.Reason.StartsWith("Retry attempt"))
        {
            transaction.AddToSet("retries", context.BackgroundJob.Id);
        }
    }

    public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        if (context.OldStateName == ScheduledState.StateName)
        {
            transaction.RemoveFromSet("retries", context.BackgroundJob.Id);
        }
    }

    public void OnStateElection(ElectStateContext context)
    {
        var failedState = context.CandidateState as FailedState;
        if (failedState == null)
        {
            // This filter accepts only failed job state.
            return;
        }

        var retryAttempt = context.GetJobParameter<int>("RetryCount") + 1;

        if (retryAttempt <= Attempts)
        {
            ScheduleAgainLater(context, retryAttempt, failedState);
        }
        else
        {
            _logger.ErrorException($"Failed to process the job '{context.BackgroundJob.Id}': an exception occurred.", failedState.Exception);
        }
    }

    private void ScheduleAgainLater(ElectStateContext context, int retryAttempt, FailedState failedState)
    {
        context.SetJobParameter("RetryCount", retryAttempt);

        const int maxMessageLength = 50;
        var exceptionMessage = failedState.Exception.Message.Length > maxMessageLength
            ? failedState.Exception.Message.Substring(0, maxMessageLength - 1) + "…"
            : failedState.Exception.Message;

        // If attempt number is less than max attempts, we should
        // schedule the job to run again later.

        var reason = $"Retry attempt {retryAttempt} of {Attempts}: {exceptionMessage}";

        context.CandidateState = (IState)new EnqueuedState
        {
            Reason = reason,
            Queue = String.Format(Queue, context.BackgroundJob.Job.Args.ToArray())
        };

        _logger.WarnException($"Failed to process the job '{context.BackgroundJob.Id}': an exception occurred. Retry attempt {retryAttempt} of {Attempts} will be performed.", failedState.Exception);
    }
}

`

@odinserj did you see this suggestion?

3 years ago, we went with the shared code-space but it can no longer be maintained, due to expanding projects are starting to overlap. A permanent solution would be quite nice, we're running 1.8.14 and the problem is still very much real :)