sensu / sensu-chef

Sensu Chef cookbook.
https://supermarket.chef.io/cookbooks/sensu
Apache License 2.0
222 stars 280 forks source link

Rough upgrade from sysv to systemd initialization #549

Closed julian7 closed 7 years ago

julian7 commented 7 years ago

When you upgrade from 0.26.5 to a higher version on a systemd-capable linux (ubuntu xenial tested), the upgrade doesn't stop old services, therefore new services will never start.

Expected Behavior

Upgrade happens automatically, with no manual intervention required. SystemV services are stopped before upgrade (maybe the correct behavior belongs to sensu project instead, and systemd setups would kill rogue processes before starting)

Current Behavior

Old processes linger, leaving systemd units in activating state:

root@luciuscornelius:~# ps -ef | grep sensu
root     13872 10932  0 08:55 pts/0    00:00:00 grep --color=auto sensu
sensu    22239     1  0 Mar14 ?        00:00:33 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/sensu-client.log -L info
root@luciuscornelius:~# less /var/log/sensu/sensu-client.log
root@luciuscornelius:~# systemctl status sensu-client
● sensu-client.service - sensu client
   Loaded: loaded (/etc/systemd/system/sensu-client.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2017-03-15 08:55:11 CET; 43s ago
  Process: 13481 ExecStart=/opt/sensu/bin/sensu-service start client (code=exited, status=1/FAILURE)
 Main PID: 13481 (code=exited, status=1/FAILURE)

Mar 15 08:55:11 luciuscornelius systemd[1]: sensu-client.service: Unit entered failed state.
Mar 15 08:55:11 luciuscornelius systemd[1]: sensu-client.service: Failed with result 'exit-code'.
root@luciuscornelius:~# killall sensu-client
root@luciuscornelius:~# ps -ef | grep sensu
root     13895 10932  0 08:56 pts/0    00:00:00 grep --color=auto sensu
root@luciuscornelius:~# systemctl restart sensu-client
root@luciuscornelius:~# systemctl status sensu-client
● sensu-client.service - sensu client
   Loaded: loaded (/etc/systemd/system/sensu-client.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-15 08:56:16 CET; 8min ago
 Main PID: 13914 (sensu-client)
    Tasks: 24 (limit: 4915)
   Memory: 21.3M
      CPU: 630ms
   CGroup: /system.slice/sensu-client.service
           └─13914 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/se

Mar 15 08:56:16 luciuscornelius systemd[1]: Started sensu client.

Possible Solution

I think of two possible resolutions of the problem:

Steps to Reproduce (for bugs)

  1. take a systemd-enabled linux system
  2. install 0.26.5-2
  3. use cookbook to upgrade it to later versions

Your Environment

cwjohnston commented 7 years ago

Hi @julian7, thanks for opening this issue. I think this is an condition that needs to be addressed outside the cookbook. I've amended the changelog in #550 to more explicitly describe this potential condition as an important upgrade consideration.