Change ssm-client log output

[gordan-bobic]

Currently ssm client services do logging by redirecting stdout to a file. This is inelegant and causes problems for log rotation. Modify the systemd service scripts to make them output to syslog:

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=<service-name>

Maybe in rpm packages at least (not sure if .deb distros use rsyslog), add a config file for rsyslog in /etc/rsyslog.d/ that makes the log output for the SyslogIdentifier actually go to the same logs where the output is redirected now.

And add a log rotation config to /etc/logrotate.d/ to rotate those logs daily.

That would make everything consistent with how it is happening now, only it would be more reliable in terms of keeping log sizes under control and not ending up with too many logs.

This will require a pre/post install script (or a change to an existing one) that modifies the existing systemd service scripts since they contain some things that are configured at ssm-admin config/add time (bind IP/port for node_exporter, database credentials for mysqld_exporter (we really should have that in a config file!)).

[oblitorum]

ssm-client is used in both client side and server side, but server side uses supervisord to manage those services, should we change it to systemd on server side so that this patch works for server side too? Or we just leave the way it is on server side?

[gordan-bobic]

I don't think systemd works properly in docker containers. So I think we should leave the server side as is.

[oblitorum]

Agreed

[oblitorum]

I found that pmm-client (current ssm-client) supports adding services for other hosts, only need to add a alias on the end of the command, e.g. ssm-admin add mysql [other flags] lore to add mysql services for lore. I presume we don't need that right? Otherwise we have to generate dynamic systemd unit files and separate config files.

PS: Working on the patches for .deb distros now, it seems that they use rsyslog too (if they use systemd). But some early versions use SysVinit system, and seems that rsyslog is not used by default, not sure if it's possible to setup the rsyslog automatically, if not we might have to redirect the output to log file manually.

[gordan-bobic]

It could be handy for cases where we run mysqld_multi to run multiple mysqld instances on a single server. In that case, we could use a template systemd service, e.g.: ssm-mysql-metrics@.service and have a config file for each alias.

When we start ssm-mysql-metrics@lore, it would read lore.conf (or lore.yml, or whatever we use for config) and start the service for that instance based on the config.

What do you think about that approach? We could keep the configs in /opt/ss/ssm/etc/mysql-metrics or /etc/ssm/mysql-metrics/ and maybe have a "default" config for most cases where we are not running multiple instances (and note that mysqld_multi configs are rare, I think I worked on maybe 3 in the last 20 years, not common at all).

So I think the feature is useful (and thus should not be removed), but on the whole I am not too concerned about it at the moment.

Also note that in cases where multiple services are running, they would each have to run on a different port.

[oblitorum]

OK, using a template systemd service seems a good approach for this case. But if we use a default config, then we have to pass the port as a command-line argument, which will break the template systemd service.

And note that there is already a /opt/ss/ssm-client directory, I think maybe we should put the configs in /opt/ss/ssm-client/xxx-metrics (I write the patch for other services too, I hope that is OK)

[gordan-bobic]

Yes, we can put the configs in there.

Default ports should not change.

For multi-service templates - I am not sure what the optimal solution would be. Maybe we do need to add an option for passing a port then. How does ssm-server do it when it runs multiple mysqld_exporters for different servers it monitors? Does it do something reasonably sensible that is portable to the client case?

Also, just to re-stress the point - the mysqld_multi case for explicitly passing aliases is really narrow. It is a rare setup. It is not justified to get bogged down by it at this point. I think we should purely focus on the default case. We can add the muti/template feature in a future release.

[oblitorum]

mysql_exporter program already accept a port argument, ssm-server runs multiple mysqld_exporters with different ports, port is passed as a command-line arguement.

Then we just ignore the multi case for now? At this point, it only supports one mysql-metrics service.

[gordan-bobic]

Yes, I think we can ignore the case on client side for now. Cool feature to add later on for the handful of customers we might get that need it, but right now I don't think it justifies the effort. We have more important features to add.

[oblitorum]

OK, got it