Ensure output from node-notifier build failures is easy to retreive

[Krinkle]

When a webhook fires and rebuilds a content site like api.jquery.com, and it a command like npm install or grunt deploy failures, the output of those commands should be easy to retrieve from the builder host. For example, via /var/log/syslog, or via sudo journalctl -u notifier-server -f -n100.

The last time I tried this, however, the output was not readily available. It seems either the appropiate log levels are turned off, or perhaps the (sub?) process output isn't captured at all from the systemd unit perspective.

Some of the past findings from ~2021 are saved at https://github.com/jquery/infrastructure/wiki/Builder-host.

[mgol]

I vaguely remember we’re using the modern-syslog package in some repos but it’s marked as an optional dependency and it fails to install due to too old Node.JS that we cannot upgrade on these hosts.

[Krinkle]

I think we can decouple the syslog use case from the Node.js code. This is, I think, how most things end up in syslog. They go there not directly from the program, but as result of the program being run as a service with its output channeled there accordingly by Linux (e.g. systemd).

That has the benefit of removing the need for configuration options, npm modules, and makes ad-hoc running and testing easier by not going to syslog in those cases. Assuming we can correclty capture and forward the exec/subprocess output in node-notifier, then the Node.js code only needs to output it using the built-in process.stdout/stderr (or console.log/error).

When we run it as a Linux service, the process runs "headless" with the output streams received by systemd and its journalctl. Tha suffices and is easy "enough" I think, to read by running journalctl. It takes (optional) params for how much and which services you want to read the output of, but the default is as good (or as bad/noisy) as e.g. var /var/log/syslog would be. I believe there is also an option in systemd (which tells Linux to run the node process forever) to enable a forwarding from the default journalctl to a syslog file.

In the old Puppet repo, we actually used this for the gruntjscom nodejs service:

[Unit]
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=gruntjscom

[Krinkle]

Solved in the new infra. node-notifier now runs with systemd, and in verbose "debug" mode, thus making build output easy to retrieve via journalctl.

https://github.com/jquery/infrastructure-puppet/commit/1d99e695b5c8ab08798ec99670a124af7862834c