Use a unique queue per sidekiq instance and only process from that queue.

[jkogara]

This approach tries to resolves the issue where multiple containers are running sidekiq in a given cluster but each instance processing a single queue that is specific to its instance and ensuring that the instance only requeues on success to that same queue.

[jkogara]

@arturictus What do you think of this change?

[arturictus]

Hi @jkogara, That sounds great! I'll check if the initial approach works. This way there is always the option to check if you want your sidekiq_alive just working or n instances working. As well it improves to only restart 1 instance if is not working instead of all the replicas at the same time. Good thinking. Sorry it toke me so long for me to answer

[arturictus]

Another think to consider is: Right now checks that 'default' queue is running. Considering that default is the queue that process most of the jobs it checks the most busy queue is running. If a wrong queue configuration is placed and the only queue activated is the one that checks the liveness it could lead to think your sidekiq is running but the only queue running is the liveness one. What do you think about @jkogara ? That doesn't mean that having the option to check multiple instances is not a great idea.

[arturictus]

Maybe we can think a different approach to achieve the same result. We store in redis which instance is running and in the server we only check for this one. example: redis keys:

• SIDEKIQ_ALIVE::INSTANCE_1
• SIDEKIQ_ALIVE::INSTANCE_2

now the problem comes: how do we get the INSTANCE_1?

[arturictus]

I think I found the solution. we can check HOSTNAME.

• in the SidekiqAlive.start we schedule the first job.

  SidekiqAlive::Worker.perform_later(ENV['HOSTNAME'])

• in the sidekiq quiet or stop we remove all the scheduled jobs with my instance HOSTNAME
• the job and the callback should be only executed if the HOSTNAME is mine if not reschedule now with the HOSTNAME received. (that could be a little bit random 😨 )
• and in the alive check we check "SIDEKIQ_ALIVE::#{ENV['HOSTNAME']" is mine.

That should make all the instances independent from each other without configuration and dynamically generated. We could make the executing queue optional with a ENV variable

what do you think @jkogara?, will this work?

[ramontayag]

In Kubernetes, how would you setup the hostname to be different per replica?

[arturictus]

Kubernetes sets ENV variable HOSTNAME per pod. We do not have to do anything.

[arturictus]

Done in #5