arturictus / sidekiq_alive

Liveness probe for Sidekiq in Kubernetes deployments
MIT License
189 stars 57 forks source link

Additional sidekiq runner child process was spawned on fork #58

Closed khun84 closed 1 year ago

khun84 commented 2 years ago

Description

We run 2 processes with different queue for the same app in a node. The capistrano config is as followed.

# config/deploy.rb
set :sidekiq_config, -> { File.join(shared_path, 'config', 'sidekiq.yml') }
set :sidekiq_default_processes, 2
set :sidekiq_processes, fetch(:sidekiq_default_processes)

set :sidekiq_options_per_process, [
  "--queue cable",
  "--queue bots --queue otps --queue default"
]
## config/sidekiq.yml

---
:verbose: true
:logfile: ./log/sidekiq.log
:concurrency: 8
:pidfile: tmp/pids/sidekiq.pid
:queues:
  - ['cable', 4]
  - ['bots', 3]
  - ['otps', 3]
  - default

When I check the process count for sidekiq, I found 2 processes (say sidekiq-0 and sidekiq-1) shortly after the deployment. However, after a while (less than 1 minute), I see another child process being spawned as well. I understand this library fork another process to run the health check server (we configure it as puma), but I expect only the puma process to be direct descendent of sidekiq-x. But what actually happen is there's an additional sidekiq runner child process (pid 7959) being spawned before the puma process was spawn (see the attached image). I also ended up with 3 pid files, sidekiq-0.pid, sidekiq-1.pid and sidekiq.pid (I don't expect this pid file to appear).

WhatsApp Image 2021-11-03 at 15 48 13

01:28 sidekiq:start
      01 RBENV_ROOT=/opt/rbenv RBENV_VERSION=2.5.9 /opt/rbenv/bin/rbenv exec bundle exec sidekiq --index 0 --pidfile /var/www/ada_staging/shared/tmp/pids/sidekiq-0.pid --environment staging --logfile /var/www/ada_staging/shared/log/sid…
    ✔ 01 deploy@xxx.xxx.xxx.xxx 0.903s
      02 RBENV_ROOT=/opt/rbenv RBENV_VERSION=2.5.9 /opt/rbenv/bin/rbenv exec bundle exec sidekiq --index 1 --pidfile /var/www/ada_staging/shared/tmp/pids/sidekiq-1.pid --environment staging --logfile /var/www/ada_staging/shared/log/sid…
    ✔ 02 deploy@xxx.xxx.xxx.xxx 0.877s

In our another environment, we only run 1 sidekiq process per app per node with sidekiq-alive enabled, the process list seems fine.

my_init,1 -u /sbin/my_init
  |-runsvdir,854 -P /etc/service
  |   |-runsv,858 sidekiq
  |   |   `-ruby,861
  |   |       `-ruby,1266
  |   |-runsv,859 sshd
  |   `-runsv,860 nginx
  `-syslog-ng,13 --pidfile /var/run/syslog-ng.pid -F --no-caps
app        861  0.9  4.3 1313672 358632 ?      Sl   02:32   3:42 sidekiq 4.2.10 webapp [0 of 7 busy] leader
app       1266  0.0  3.2 979984 269032 ?       Sl   02:33   0:02 puma 3.7.1 (tcp://0.0.0.0:7433) [webapp]

The mentioned behaviour does not happen if I disable sidekiq-alive altogether. Does that mean our sidekiq process setup is not suitable to use sidekiq-alive?

Environment

  1. ubuntu-16.04
  2. ruby 2.5.9
  3. rails 5.1.7
  4. sidekiq (4.2.10)
  5. sidekiq-ent (1.6.1)
  6. sidekiq-pro (3.7.1)
  7. sidekiq_alive (2.1.4)
  8. capistrano-sidekiq (1.0.3)