Sidecar Idea: pre-validation

[lennartkoopmann]

I'm facing the following problem: I'm managing auditd and rsyslogd through sidecar so, to run, these services have to be disabled and no longer be managed by systemd.

I don't have configuration management, so I can't just quickly roll-out a change to install sidecar everywhere, and in the same moment, also disable the services.

On some hosts, I will forget to disable the services, and the sidecar-managed processes will fail.

So here is my idea: Could there be a "pre-validation" step that executes and checks the output or response to a small script? For rsyslogd to be started using sidecar, I'd check first what systemctl status rsyslogd returns and put the process in a "pre-validation failed: rsyslogd not disabled in systemd" or something mode. This way, it is more obvious to me what happened. Especially compared to the generic "failing" status. I could also use it to check that a package is installed or a file exists, etc.

[bernd]

I am not sure if I would let the sidecar take over the management of systemd/init.d/upstart managed processes. This would open up a big can of worms, IMHO. :wink:

As far as I know, nothing prevents you from running another instance of sidecar managed rsyslog or auditd. I am doing this with rsyslog already. (e.g. -n -f %s -i /run/sidecar-rsyslog.pid) Yes, you will waste some resources but then you don't have to touch the system config of these processes.

Also, if we should decide to put all generated config files into one directory under /var/lib/graylog-sidecar, we cannot overwrite arbitrary config files in /etc anymore. (which I think is a good thing) See https://github.com/Graylog2/graylog2-server/issues/4887 for details.

If we really want to let sidecar take over management of system processes, we would need even tighter integration with systemd and maybe even other service managers. (e.g. windows) This requires careful evaluation and discussion.

[lennartkoopmann]

Thanks for the reply! Maybe you misunderstood me: I'm not suggesting that Sidecar would manage system processes in any way, but that it could check if certain conditions are met before attempting to start a process.

For example, auditd can only run once per system and is enabled as a systemd process by default after installation. All the sidecar would do, is to check if it's still enabled by parsing the output of systemctl status auditd. This would also be a user-supplied script and not something that sidecar does out of the box.

What do you think?

[bernd]

I think I mixed up the facts from this issue and the ones in #5049 about installing a generated config file in /etc/rsyslog.d and thought, that this is all about supporting the management of system services. Sorry about that!

I am still not sure if this should be the responsibility of the sidecar. If we decide it is, I think we should implement better integration with systemd and also windows services to handle issues like the one you described with auditd. (first thing that comes to mind: being able to configure conflicting services. but there will be other things) Having a pre-validation command sounds like a quick solution to the problem, but I am afraid of how this can and will be used. :smile:

If we go the pre-validation command route, we have to make sure that this cannot be used to execute arbitrary scripts on the sidecar node. Basically the same we do for the collector binaries. (white list for scripts etc)

[lennartkoopmann]

Very good point regarding the whitelist and safety. Let's discuss on Wednesday, but I think you just convinced me not to do it. :D