Impossible to run in Kubernetes properly

egopher commented 5 years ago

Hi guys. I want to run ASP.NET Core and schedule background tasks with hangfire. However for some reason it kills the main process (with PID 1 in container) and thus Kubernetes "thinks" that the container has crashed and sends sigterm.

mike-dube commented 5 years ago

Please provide some source code.

Hangfire can be implemented by various way.

On Kubernetes sides: ensure your pod is not being killed because of probe failling. You can use kubectl describe.

odinserj commented 5 years ago

Hi @egopher. Do you have any logs from Hangfire or crash dump?

egopher commented 5 years ago

Liveness probe is present and works

livenessProbe: httpGet: port:80 path: /health initialDelaySeconds: 60 periodSeconds: 60

The health probe works as intended. However, it seems like hangfire core kills the main thread which has pid 1 thus kubernetes scheduler makes decision to SIGTERM the container and restart it.

mike-dube commented 5 years ago

As @odinserj said, do you have the logs on the exception? If Hangfire throws 1, there's probably an underlying exception that block the WebHost builder to start correctly.

MikelThief commented 5 years ago

I cannot confirm that this happens. Hangfire works for me in Kubernetes.

saliehendricks commented 3 years ago

@odinserj @dekim I have Hangfire working in Kubernetes (AKS) in ASP.Net Core 3.1, however it seems to shut down when it runs as a Console in another container. Below is my logs from Kubernetes and my startup code. Kubernetes has an error about Console.ReadKey since the Console is being intercepted. Recommends I use Read instead. Perhaps the issue is how to keep the container alive in Kubernetes?

My logs for the console:

2020-11-25T14:42:44.165604916Z>>Processor is being configured..
2020-11-25T14:42:44.478576198Z Using HF REDIS connection: xxxx
2020-11-25T14:42:44.677940382Z Hangfire Server Starting
2020-11-25T14:42:44.993191521Z
2020-11-25 02:42:44 [INFO] (Hangfire.Pro.Redis.RedisStorage) Connection to Redis server(s) was successfully established.
2020-11-25T14:42:44.995102652Z
2020-11-25 02:42:44 [INFO] (Hangfire.BackgroundJobServer) Starting Hangfire Server using job storage: 'redis://hapi-back:6379/0'
2020-11-25T14:42:44.995265063Z
2020-11-25 02:42:44 [INFO] (Hangfire.BackgroundJobServer) Using the following options for Hangfire Server:
2020-11-25T14:42:44.995271564Z Worker count: 10
2020-11-25T14:42:44.995274864Z Listening queues: 'default'
2020-11-25T14:42:44.995278164Z Shutdown timeout: 00:00:15
2020-11-25T14:42:44.995281264Z Schedule polling interval: 00:00:15
2020-11-25T14:42:45.053581466ZHangfire Server started. Press any key to exit...
2020-11-25T14:42:45.055948628Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 caught stopping signal...
2020-11-25T14:42:45.056026834Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 caught stopped signal...
2020-11-25T14:42:45.165917077Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 successfully announced in 108.6463 ms
2020-11-25T14:42:45.169426717Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, FetchedJobsWatcher, Worker, DelayedJobScheduler, RecurringJobScheduler...
2020-11-25T14:42:45.180015444Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 all the dispatchers started
2020-11-25T14:42:45.183921312Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 All dispatchers stopped
2020-11-25T14:42:45.751284555Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 successfully reported itself as stopped in 564.8575 ms
2020-11-25T14:42:45.751321558Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 has been stopped in total 695.6778 ms

[REPEATED..]
2020-11-25T14:45:41.873864566Z>>Processor is being configured..
...

My consoles startup code:

public static void Main(string[] args)
{
           Console.WriteLine(">>Processor is being configured..");
           //...
           var options = new RedisStorageOptions
            {
                Prefix = "hangfire-hapi:",
                InvisibilityTimeout = TimeSpan.FromHours(3),
            };

            GlobalConfiguration.Configuration
                .UseAutofacActivator(services.Build(), false)
                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                .UseColouredConsoleLogProvider()
                .UseSimpleAssemblyNameTypeSerializer()
                .UseRecommendedSerializerSettings()
                .UseRedisStorage(hangfireConnection,options);

            BackgroundJobServerOptions bjop = new BackgroundJobServerOptions()
            {
                WorkerCount = 10,

            };

            Console.WriteLine("Hangfire Server Starting");
            using (var server = new BackgroundJobServer(bjop))
            {                
                Console.WriteLine("Hangfire Server started. Press any key to exit...");
                Console.Read(); // Kubernetes throws an error when using ReadKey()
            }

}

mike-dube commented 3 years ago

@odinserj @dekim I have Hangfire working in Kubernetes (AKS) in ASP.Net Core 3.1, however it seems to shut down when it runs as a Console in another container. Below is my logs from Kubernetes and my startup code. Kubernetes has an error about Console.ReadKey since the Console is being intercepted. Recommends I use Read instead. Perhaps the issue is how to keep the container alive in Kubernetes?

My logs for the console:

2020-11-25T14:42:44.165604916Z>>Processor is being configured..
2020-11-25T14:42:44.478576198Z Using HF REDIS connection: xxxx
2020-11-25T14:42:44.677940382Z Hangfire Server Starting
2020-11-25T14:42:44.993191521Z
2020-11-25 02:42:44 [INFO] (Hangfire.Pro.Redis.RedisStorage) Connection to Redis server(s) was successfully established.
2020-11-25T14:42:44.995102652Z
2020-11-25 02:42:44 [INFO] (Hangfire.BackgroundJobServer) Starting Hangfire Server using job storage: 'redis://hapi-back:6379/0'
2020-11-25T14:42:44.995265063Z
2020-11-25 02:42:44 [INFO] (Hangfire.BackgroundJobServer) Using the following options for Hangfire Server:
2020-11-25T14:42:44.995271564Z Worker count: 10
2020-11-25T14:42:44.995274864Z Listening queues: 'default'
2020-11-25T14:42:44.995278164Z Shutdown timeout: 00:00:15
2020-11-25T14:42:44.995281264Z Schedule polling interval: 00:00:15
2020-11-25T14:42:45.053581466ZHangfire Server started. Press any key to exit...
2020-11-25T14:42:45.055948628Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 caught stopping signal...
2020-11-25T14:42:45.056026834Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 caught stopped signal...
2020-11-25T14:42:45.165917077Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 successfully announced in 108.6463 ms
2020-11-25T14:42:45.169426717Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, FetchedJobsWatcher, Worker, DelayedJobScheduler, RecurringJobScheduler...
2020-11-25T14:42:45.180015444Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 all the dispatchers started
2020-11-25T14:42:45.183921312Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 All dispatchers stopped
2020-11-25T14:42:45.751284555Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 successfully reported itself as stopped in 564.8575 ms
2020-11-25T14:42:45.751321558Z
2020-11-25 02:42:45 [INFO] (Hangfire.Server.BackgroundServerProcess) Server hapi-processor-55cd6fb64-qzjh2:1:3a46a065 has been stopped in total 695.6778 ms

[REPEATED..]
2020-11-25T14:45:41.873864566Z>>Processor is being configured..
...

My consoles startup code:

public static void Main(string[] args)
{
           Console.WriteLine(">>Processor is being configured..");
           //...
           var options = new RedisStorageOptions
            {
                Prefix = "hangfire-hapi:",
                InvisibilityTimeout = TimeSpan.FromHours(3),
            };

            GlobalConfiguration.Configuration
                .UseAutofacActivator(services.Build(), false)
                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                .UseColouredConsoleLogProvider()
                .UseSimpleAssemblyNameTypeSerializer()
                .UseRecommendedSerializerSettings()
                .UseRedisStorage(hangfireConnection,options);

            BackgroundJobServerOptions bjop = new BackgroundJobServerOptions()
            {
                WorkerCount = 10,

            };

            Console.WriteLine("Hangfire Server Starting");
            using (var server = new BackgroundJobServer(bjop))
            {                
                Console.WriteLine("Hangfire Server started. Press any key to exit...");
                Console.Read(); // Kubernetes throws an error when using ReadKey()
            }

}

Replace Console.Read by a while(true) and you should be good to go!

HangfireIO / Hangfire

Impossible to run in Kubernetes properly #1483