search

lobsters / lobsters-ansible

Ansible playbook for lobste.rs
ISC License
79 stars 26 forks source link

systemd doesn't recognize unicorn restarted successfully #29

Closed pushcx closed 6 years ago

pushcx commented 6 years ago

The new unicorn systemd service works correctly to restart unicorn on deploy but reports failure:

TASK [lobsters : restart unicorn] ******************************************************************************************************************************************************************
fatal: [lobsters.xen.prgmr.com]: FAILED! => {"changed": false, "msg": "Unable to reload service lobsters-unicorn: Job for lobsters-unicorn.service failed because the control process exited with error code. See \"systemctl status lobsters-unicorn.service\" and \"journalctl -xe\" for details.\n"}
        to retry, use: --limit @/home/pushcx/code/l-ansible/prod.retry
● lobsters-unicorn.service - Lobsters Unicorn Server                                                                                                                                               
   Loaded: loaded (/etc/systemd/system/lobsters-unicorn.service; enabled; vendor preset: enabled)                                                                                                  
   Active: active (running) (Result: exit-code) since Wed 2018-05-02 14:10:05 UTC; 20min ago
  Process: 7278 ExecStop=/usr/bin/kill -QUIT $MAINPID (code=exited, status=203/EXEC)
  Process: 9439 ExecReload=/usr/bin/kill -USR2 $MAINPID (code=exited, status=203/EXEC)
 Main PID: 7290 (ruby2.3)
   CGroup: /system.slice/lobsters-unicorn.service
           ├─7290 unicorn_rails master -c config/unicorn.conf.rb -E production -D config.ru                                                                                                        
           ├─7376 unicorn_rails worker[0] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7381 unicorn_rails worker[1] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7387 unicorn_rails worker[2] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7392 unicorn_rails worker[3] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7399 unicorn_rails worker[4] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7404 unicorn_rails worker[5] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7409 unicorn_rails worker[6] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7415 unicorn_rails worker[7] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7420 unicorn_rails worker[8] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7426 unicorn_rails worker[9] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                     
           ├─7431 unicorn_rails worker[10] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                    
           └─7436 unicorn_rails worker[11] -c config/unicorn.conf.rb -E production -D config.ru                                                                                                    

May 02 14:10:05 lobsters systemd[1]: Stopped Lobsters Unicorn Server.
May 02 14:10:05 lobsters systemd[1]: lobsters-unicorn.service: Unit entered failed state.
May 02 14:10:05 lobsters systemd[1]: lobsters-unicorn.service: Failed with result 'exit-code'.
May 02 14:10:05 lobsters systemd[1]: Started Lobsters Unicorn Server.
May 02 14:22:20 lobsters systemd[1]: Reloading Lobsters Unicorn Server.
May 02 14:22:20 lobsters systemd[1]: lobsters-unicorn.service: Control process exited, code=exited status=203
May 02 14:22:20 lobsters systemd[1]: Reload failed for Lobsters Unicorn Server.
May 02 14:25:22 lobsters systemd[1]: Reloading Lobsters Unicorn Server.
May 02 14:25:22 lobsters systemd[1]: lobsters-unicorn.service: Control process exited, code=exited status=203
May 02 14:25:22 lobsters systemd[1]: Reload failed for Lobsters Unicorn Server.
--                                                                                                                                                                                                  
-- The error number returned by this process is 2.
May 02 14:25:22 lobsters systemd[1]: lobsters-unicorn.service: Control process exited, code=exited status=203                                                                                      
May 02 14:25:22 lobsters systemd[1]: Reload failed for Lobsters Unicorn Server.
-- Subject: Unit lobsters-unicorn.service has finished reloading its configuration
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit lobsters-unicorn.service has finished reloading its configuration
--
-- The result is failed.

My reading is that systemd thinks unicorn has crashed when the unicorn master and/or worker processes exit with with code 203 to indicate they're restarting. Can we configure the service to consider that a clean restart?

Tagging contributors to #23 @jstoja @PeterFaiman because y'all obviously know systemd better than me.

jstoja commented 6 years ago

Looking at Systemd's exit codes, this means that the reload task has an ENOENT on some execve call.

I found that it's the /usr/bin/kill that doesn't exist at this path on ubuntu... (and on most distros). I'm comming with a PR to solve this. I've also digged into Unicorn and Systemd and it seems that the reload trick in unicorn's configuration is helping systemd a lot!