st2 services won't start due to wrong configuration path

[jvrplmlmn]

Using the latest version of this cookbook, on CentOS 7.2, with StackStorm 2.0.0, results in most st2* services not starting up. The /etc/st2/st2.conf file is missing several options.

Most references to the logging configuration files are wrong, since they point to /etc/st2api/... and at least with 2.0.0 the configuration files are in /etc/st2/...:

[st2@localhost ~]$ DAEMON_ARGS="-k eventlet -b 127.0.0.1:9101 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30"
[st2@localhost ~]$ /opt/stackstorm/st2/bin/gunicorn_pecan /opt/stackstorm/st2/lib/python2.7/site-packages/st2api/gunicorn_config.py $DAEMON_ARGS
[2016-09-06 08:31:58 +0000] [24124] [INFO] Starting gunicorn 19.6.0
[2016-09-06 08:31:58 +0000] [24124] [INFO] Listening at: http://127.0.0.1:9101 (24124)
[2016-09-06 08:31:58 +0000] [24124] [INFO] Using worker: eventlet
[2016-09-06 08:31:58 +0000] [24172] [INFO] Booting worker with pid: 24172
2016-09-06 08:31:59,122 DEBUG [-] Using config files: /etc/st2/st2.conf
2016-09-06 08:31:59,122 DEBUG [-] Using logging config: /etc/st2api/console.conf
ERROR: Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/log.py", line 185, in setup
    disable_existing_loggers=disable_existing_loggers)
  File "/usr/lib64/python2.7/logging/config.py", line 70, in fileConfig
    formatters = _create_formatters(cp)
  File "/usr/lib64/python2.7/logging/config.py", line 106, in _create_formatters
    flist = cp.get("formatters", "keys")
  File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
NoSectionError: No section: 'formatters'
[2016-09-06 08:31:59 +0000] [24172] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
    worker.init_process()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 96, in init_process
    super(EventletWorker, self).init_process()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/pecan/commands/serve.py", line 190, in load
    return deploy(self.cfgfname)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/pecan/deploy.py", line 9, in deploy
    return load_app(config)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/pecan/core.py", line 197, in load_app
    app = module.app.setup_app(_runtime_conf, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/app.py", line 70, in setup_app
    config_args=config.config_args)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/service_setup.py", line 88, in setup
    excludes=cfg.CONF.log.excludes)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/log.py", line 195, in setup
    raise Exception(six.text_type(exc))
Exception: No section: 'formatters'
[2016-09-06 08:31:59 +0000] [24172] [INFO] Worker exiting (pid: 24172)
[2016-09-06 08:31:59 +0000] [24124] [INFO] Shutting down: Master
[2016-09-06 08:31:59 +0000] [24124] [INFO] Reason: Worker failed to boot.
[st2@localhost ~]$

Probably the packaging has changed where the configuration files are created.

[arm4b]

Looks like templates/default/st2.conf.erb is outdated and unsync from the original st2.conf file.

Here is the default config /etc/st2/st2.conf which is shipped with packaging: https://github.com/StackStorm/st2/blob/master/conf/st2.prod.conf

[arm4b]

Apart of that, I'm :-1: with the templating approach on Chef side (https://github.com/StackStorm/chef-stackstorm/blob/master/templates/default/st2.conf.erb).

Instead of templating, chef-stackstorm should just use original st2.conf installed by default via packaging. If user needs to overwrite some specific conf settings - chef cookbook should edit the default file /etc/st2/st2/conf.

Otherwise it will always go out of sync, for example when StackStorm core team changes some recommended defaults in st2.conf.

[jvrplmlmn]

chef cookbook should edit the default file /etc/st2/st2/conf.

Sorry if I'm not following @armab, but you mean /etc/st2/st2.conf ?

[arm4b]

@jvrplmlmn You're right, /etc/st2/st2.conf. Sorry for the typo.

[jvrplmlmn]

But in that case, if a user of the cookbook wants to overwrite a value, they need to use their own file / template and override the default one. Or which alternative are you proposing? Using something like crudini?

[jvrplmlmn]

I checked that every st2* service definition has a EnvironmentFile like:

EnvironmentFile=-/etc/sysconfig/st2notifier

Is it possible to override those default values in this files?

[arm4b]

crudini

Yeah, something like that. For Puppet there is puppetlabs/inifile, probably something similar exists for Chef. IDK.

[arm4b]

Actually, nevermind.

I just checked that https://github.com/StackStorm/chef-stackstorm/blob/master/templates/default/st2.conf.erb is 1 year old (!), so it's from very very old StackStorm installation (before we even introduced new packages).

So just sync-ing default values in https://github.com/StackStorm/chef-stackstorm/blob/master/templates/default/st2.conf.erb and https://github.com/StackStorm/st2/blob/master/conf/st2.package.conf should be good and easy enough.

Thinking about the balance and not messing with Chef practices