search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.05k stars 745 forks source link

st2api service fails to restart after adding configuration for backend redis service #2640

Open stevemuskiewicz opened 8 years ago

stevemuskiewicz commented 8 years ago

Using the Stackstorm 1.3 package-based install, I went in and configured redis in /etc/st2/st2.conf per the policies docs. This seemed to be fine initially but eventually something must have happened to cause the st2api service to restart and then (according to systemctl -l status st2api.service output) it looks like it got into some kind of restart loop.

I finally caught the offending output (this was pretty difficult to do and doesn't appear to be making its way into st2api.log), looks like the /opt/stackstorm/st2 Python virtualenv was missing the redis module:

st2api.service - StackStorm service st2api Loaded: loaded (/usr/lib/systemd/system/st2api.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2016-04-21 10:42:01 EDT; 28s ago Main PID: 26270 (gunicorn_pecan) CGroup: /system.slice/st2api.service \u251c\u250026270 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn_pecan /opt/stackstorm/st2/lib/python2.7/site-packages/st2api/gunicorn_config.py -k eventlet -b 127.0.0.1:9101 --threads 10 --workers 1 --graceful-timeout 10 \u2514\u250026522 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn_pecan /opt/stackstorm/st2/lib/python2.7/site-packages/st2api/gunicorn_config.py -k eventlet -b 127.0.0.1:9101 --threads 10 --workers 1 --graceful-timeout 10

Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: verify_requirements, Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/stevedore/extension.py", line 183, in _load_one_plugin Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: plugin = ep.resolve() Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/pkg_resources/init.py", line 2208, in resolve Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: module = import(self.module_name, fromlist=['name'], level=0) Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/tooz/drivers/redis.py", line 27, in Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: import redis Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: ImportError: No module named redis Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: [2016-04-21 10:42:30 +0000] [26522] [INFO] Worker exiting (pid: 26522)

My fix was to install the redis module into the virtualenv, ie. sudo su; source /opt/stackstorm/st2/bin/activate; pip install redis

After this systemctl restart st2api appeared to work and everything was happy again.

Not sure if this is a missing dependency in packaging or a setup script or just needs to be noted in the docs. Let me know if you need any more info about my setup.

lakshmi-kannan commented 8 years ago

There isn't much we can do other than document it. We don't want to ship all the libraries by default. That seems a little less optimal. Thanks for the steps :).

Maybe we can build some tooling around this? I am going to brainstorm internally to see how we can improve this but meanwhile here is a docs PR https://github.com/StackStorm/st2docs/pull/131

stevemuskiewicz commented 8 years ago

Works for me, thanks!