Hangfire not resume after recycle/stop and play application pool

[ghost]

Hi,

I read this article http://docs.hangfire.io/en/latest/deployment-to-production/making-aspnet-app-always-running.html

And I followed every step, but If I recycle the application pool hangfire do nothing until I make a request to the site that live in the application pool (eg. I open the browser and try to open the site)

example: I have firefox and I can see the dashboard of hangifre. mysite.mydomain/hangfire/dashboard

Application pool is up and hangfire is up. All work

Close the browser and click recycle.

I must open mysite.mydomain with firefox otherwise hangfire do nothing.

Maybe a bug?

regards,

[odinserj]

Looks like a misconfiguration. Please post here your HangfireBootstrapper.cs, Startup.cs and applicationpool.config files.

[ghost]

Hi, This is HangfireBootstrapper

public class HangfireBootstrapper : IRegisteredObject
    {
        public static readonly HangfireBootstrapper Instance = new HangfireBootstrapper();
        private readonly object _lockObject = new object();
        private bool _started;

        private BackgroundJobServer _backgroundJobServer;

        private HangfireBootstrapper()
        {
        }

        public void Start()
        {
            lock (_lockObject)
            {
                if (_started) return;
                _started = true;
                AreaRegistration.RegisterAllAreas();
                System.Web.Http.GlobalConfiguration.Configure(WebApiConfig.Register);
                RouteConfig.RegisterRoutes(RouteTable.Routes);
                AutofacConfig.Startup();
                GlobalConfiguration.Configuration.UseSqlServerStorage("TEST_DB");
                _backgroundJobServer = new BackgroundJobServer();
            }
        }

        public void Stop()
        {
            lock (_lockObject)
            {
                if (_backgroundJobServer != null)
                {
                    _backgroundJobServer.Dispose();
                }

                HostingEnvironment.UnregisterObject(this);
            }
        }

        void IRegisteredObject.Stop(bool immediate)
        {
            Stop();
        }
}

This is startup.cs

public class Startup
    {
        public void Configuration(IAppBuilder app)
        {
            app.UseHangfireDashboard("/hangfire");

            UpdateTask.StartMainTask(); //call to the method that register the background job
        }
    }

I modified the file applicationHost.config as:

<system.applicationHost>

        <applicationPools>
            <add name="DefaultAppPool" />
<!--  other application pool-->
            <add name="test-hangfire-proj" autoStart="false" managedRuntimeVersion="v4.0" startMode="AlwaysRunning">
                <processModel idleTimeout="00:00:00" idleTimeoutAction="Suspend" />
            </add>
            <applicationPoolDefaults managedRuntimeVersion="v4.0">
                <processModel identityType="ApplicationPoolIdentity" />
            </applicationPoolDefaults>
        </applicationPools>

<!-- other xml -->

<sites>
<!-- other sites-->
<site name="test-hangfire-proj" id="22" serverAutoStart="false">
                <application path="/" applicationPool="test-hangfire-proj" serviceAutoStartEnabled="true" serviceAutoStartProvider="ApplicationPreload">
                    <virtualDirectory path="/" physicalPath="D:\inetpub\wwwroot\test-hangfire-proj" />
                </application>
                <bindings>
<!-- it's an intranet site -->
                    <binding protocol="http" bindingInformation="*:80:test-hangfire-proj.local" />
                </bindings>
            </site>
</sites>
<webLimits />
    <serviceAutoStartProviders>
        <add name="ApplicationPreload" type="testHangfire.ApplicationPreload, testHangfire, Version=1.0.0.0, Culture=neutral" />
    </serviceAutoStartProviders>

    </system.applicationHost>

Thanks in advance for the help

[odinserj]

Can you show me the code behind UpdateTask.StartMainTask?

[ghost]

It's a bit complicated, I hope you can find what you need. I add some comment. This code does not throw exception and work as expected until a recycle

public static class UpdateTask
{
        private const string NAME_FAST_UPDATE = "FastUpdate";
        private const string NAME_FULL_UPDATE = "FullUpdate";

        private static readonly object LOCK_OBJ = new object();

        private const string NAME_MASTER_TASK = "master_task";
        private static readonly string PATH_CONFIG_FREQ = HostingEnvironment.MapPath("~/task_plan_eventi.config");

        private static ILogger _logger; //serilog interface
        private static IDBServices _idbServices; 

//register the main task
        public static void StartMainTask()
        {
            _idbServices = (IDBServices) DependencyResolver.Current.GetService(typeof (IDBServices));
            _logger = (ILogger) DependencyResolver.Current.GetService(typeof (ILogger));
            RecurringJob.AddOrUpdate(NAME_MASTER_TASK, () => RefreshTask(), "*/1 * * * *"); 
        }

//this code read a json (place in PATH_CONFIG_FREQ) and set two task
public static void RefreshTask()
        {

            try
            {
                if (!File.Exists(PATH_CONFIG_FREQ))
                {
                    _logger.Error("File json not present");
                    return;
                }
                var type = new
                {
                    task = new List<TaskValue>() //task value is a class with some field
                };

                List<TaskValue> freqs;
                try
                {
                    freqs = Newtonsoft.Json.JsonConvert.DeserializeAnonymousType(File.ReadAllText(PATH_CONFIG_FREQ), type)
                            .task.OrderBy(t => t.Nome)
                            .ToList();
                }
                catch (Exception exc)
                {
                    _logger.Error(exc, "Problem with json file");
                    return;
                }

                if (freqs.Count != 2)
                {
                    _logger.Error(string.Format("The value in the file must be {0} ", 2));
                    return;
                }
                if (freqs.FindIndex(f => f.Nome == NAME_FAST_UPDATE) == -1 ||
                    freqs.FindIndex(f => f.Nome == NAME_FULL_UPDATE) == -1)
                {
                    _logger.Error(string.Format("Task name not corrent must be {0} and {1} ",
                        NAME_FAST_UPDATE, NAME_FULL_UPDATE));
                    return;
                }

//in my db I have a table with the task I registered, to avoid useless update of the task
                var freqInDb = _idbServices.GetAllTaskValue();

                if (freqInDb.Count == 0)
                {
                    _idbServices.InsertAllTask(freqs);
                    foreach (var task in freqs)
                    {
                        try
                        {
// ReSharper disable AccessToForEachVariableInClosure
                            RecurringJob.AddOrUpdate(task.Nome, () => CallMethod(
                                task.Nome), task.GetCronExpr(), TimeZoneInfo.Local);
// ReSharper restore AccessToForEachVariableInClosure
                        }
                        catch (Exception exc)
                        {
                            _logger.Error(exc, string.Format("Problem with task {0} ", task.Name));
                            return;
                        }
                    }
                }
                else
                {
                    for (var i = 0; i < freqInDb.Count; ++i)
                    {
                        var taskToUpdate = freqs[i];
//the task no need to update
                        if (freqInDb[i].Equals(taskToUpdate)) continue;
                        _idbServices.UpdateTask(taskToUpdate);
                        try
                        {
                            RecurringJob.AddOrUpdate(taskToUpdate.Nome, () => CallMethod(taskToUpdate.Nome),
                                    taskToUpdate.GetCronExpr(), TimeZoneInfo.Local);
                        }
                        catch (Exception exc)
                        {
                             _logger.Error(exc, string.Format("Problem with task {0} ", task.Name));
                            return;
                        }
                    }
                }
            }
            catch (Exception exc)
            {
                _logger.Fatal(exc, "fatal error");
            }
        }

        public static void CallMethod(string nameMethod)
        {
            typeof (UpdateTask).GetMethod(nameMethod,
                BindingFlags.Public | BindingFlags.Static).Invoke(null, null);
        }

//at the moment these method print something in the log
public static void FastUpdate()
        {
                _logger.Information("Fast update start");
                lock (LOCK_OBJ)
                {
                    _logger.Information("Fast update get lock");
                 }
        }

public static void FullUpdate()
        {
                _logger.Information("full update start");
                lock (LOCK_OBJ)
                {
                    _logger.Information(" full update get lock");
                }
}

Thanks

[ghost]

Hi, No news?

[Cussa]

I have same problem here. I followed the recommended instructions, but when the application recycles, it stops the jobs.

[SergeyA]

Every time I upload a new version of my asp.net app in same directory Hangfire stops until I restart IIS. It may be related to this issue.

[odinserj]

@lfongaroScp, the code is really complex. Please reduce it to necessary minimum as it will take too long to understand and debug it.

@Cussa, @SergeyA, what stops in real, Hangfire or the whole application?

[Cussa]

In my case, the application IS the Hangfire. I create a instance only to have a way to execute async and batch code.

I configured my IIS as recomended and the application. Start that, and there is a job that is sending a email to me. After 20min, it's stops to send the e-mail. And when i open the application again, it's restart to send the e-mails.

[ghost]

@odinserj

I reduce the amount of code as possible:

public static class UpdateTask
{
        private const string NAME_FAST_UPDATE = "FastUpdate";
        private const string NAME_FULL_UPDATE = "FullUpdate";

        private static readonly object LOCK_OBJ = new object();

        private const string NAME_MASTER_TASK = "master_task";
        private static readonly string PATH_CONFIG_FREQ = HostingEnvironment.MapPath("~/task_plan_eventi.config");

        private static ILogger _logger; //serilog interface

//register the main task
        public static void StartMainTask()
        {
            _logger = (ILogger) DependencyResolver.Current.GetService(typeof (ILogger));
            RecurringJob.AddOrUpdate(NAME_MASTER_TASK, () => RefreshTask(), "*/1 * * * *"); 
        }

//this code read a json (place in PATH_CONFIG_FREQ) and set two task
public static void RefreshTask()
        {
            //code omitted......
                //get the values from a file
                foreach (var task in freqs)
                {
                    RecurringJob.AddOrUpdate(task.Nome, () => CallMethod(
                        task.Nome), task.GetCronExpr(), TimeZoneInfo.Local);
                }
        }

        public static void CallMethod(string nameMethod)
        {
            typeof (UpdateTask).GetMethod(nameMethod,
                BindingFlags.Public | BindingFlags.Static).Invoke(null, null);
        }

//at the moment these method print something in the log
public static void FastUpdate()
        {
                _logger.Information("Fast update start");
                lock (LOCK_OBJ)
                {
                    _logger.Information("Fast update get lock");
                 }
        }

public static void FullUpdate()
        {
                _logger.Information("full update start");
                lock (LOCK_OBJ)
                {
                    _logger.Information(" full update get lock");
                }
        }
}

[odinserj]

@lfongaroScp, call the following line from HangfireBootstrapper class:

UpdateTask.StartMainTask(); //call to the method that register the background job

The Configure method is called only when someone hits the application URL, so you get this behavior.

[ghost]

Hi, Thx for the reply. I add the call in HangfireBootstrapper, but It throw exception after a ricyle because DependencyResolver.Current it is null.

It seems IIS It's still down.

[odinserj]

Where DependencyResolver.Current is initialized?

[ghost]

In the Startup(). The method is called by OWIN.

[odinserj]

So it should be also initialized in HangfireBootstrapper

[waynebrantley]

Set your application pool to terminate instead of suspend

[dan91d]

I have the same problem. Any solution found?

[kvansaders]

Wouldn't creating a health check endpoint and hitting it every 5 min via powershell or curl do the trick? Why all this bootstrapper nonsense?