Disable rsyslog logging for ST2 services

sibirajal commented 6 years ago

If you find an issue in packages, please file an issue and we'll have a look as soon as we can. In order to expedite the process, it would be helpful to follow this checklist and provide relevant information.

[ CentOS Linux release 7.3.1611 (Core) ] Operating system: uname -a, ./etc/lsb_release or cat /etc/redhat-release
[st2 2.6.0 ] StackStorm version: st2 --version
[All st2 services ] Actual package versions of all packages (st2, st2web, st2chatops, st2mistral, nginx, mongo, rabbitmq-server, postrgresql; Enterprise: st2flow, st2-auth-ldap)

Issue details

I have noticed in our ST2 instance that rsyslog messages log file is flooded by st2 services. All the execution results are logged in their own st2 log files in /var/log/st2 and /var/log/messages.

Since each st2 service is configured with the dedicated log file, rsyslogd shouldn't be used for logging. This is causing disk space problem in our production instance as the logs are written redundantly in 2 places.

Can you please disable the ST2 default logging to rsyslog? This can be further manually enabled if something needed in future.

cat /etc/rsyslog.d/* /etc/rsyslog.conf | grep '/var/log/messages'
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

arm4b commented 6 years ago

Thanks @sibirajal for opening an issue!

So as discussed, the reason behind this behavior, systemd by default forwards all the messages from journalctl to syslog. Depending on rsyslogd configuration, duplicated logs from StackStorm apart of /var/log/st2/ might end in /var/log/messages or /var/log/syslog.

We can't do anything in our service unit files since it's core systemd behavior, but we can provide an example rsyslogd config which will filter out all StackStorm-related messages from appearing in syslog.

Here is an example of working /etc/rsyslog.d/33-st2.conf config:

if $programname == 'st2api' then stop
if $programname == 'st2auth' then stop
if $programname == 'st2actionrunner' then stop
if $programname == 'st2rulesengine' then stop
if $programname == 'st2sensorcontainer' then stop
if $programname == 'st2garbagecollector' then stop
if $programname == 'st2resultstracker' then stop
if $programname == 'st2stream' then stop
if $programname == 'st2notifier' then stop
if $programname == 'st2workflowengine' then stop
if $programname == 'st2scheduler' then stop
if $programname == 'st2chatops' then stop
if $programname == 'mistral-server' then stop
if $programname == 'mistral-api' then stop
if $programname == 'gunicorn' then stop

The possible resolution would be shipping that rsyslogd config via packaging, same as we do with logrotate configs.

But the decision is not that clearly beneficial. The problem is: while this is a good thing for one user, it might be a bad default for other users and we don't know yet if adding such default is 👍 or 👎 . Some users configure syslog to forward messages to external centralized log storage, some are configuring for storing in local log files or use any other custom modifications.

I'm not sure yet if we should ship our "force no-log" rsyslogd config, so I'm leaving the issue here for gathering more feedback.

vincent-legoll commented 6 years ago

I had the same problem, put the rsyslog config snippet from above in /etc/rsyslog.d, and it looks like it is working properly now.

I now only have hubot spamming my global log. See : https://github.com/StackStorm/st2-packages/issues/300.

By adding the following line:

if $programname == 'hubot' then stop

to the rsyslog config, I can get rid of those entries from the global log, but they are getting nowhere now...

Still missing the /etc/st2/logging.chatops_or_hubot.conf

NikonNLG commented 5 years ago

Two more lines if someone search for solution:

if $programname == 'st2workflowengine' then stop
if $programname == 'st2scheduler' then stop

StackStorm / st2-packages

Disable rsyslog logging for ST2 services #532

Issue details