Logging

[KillingSpark]

Rustysd needs persistent logging. This needs some questions answered:

1. Should rustysds logs go somewhere else than stdout/stderr output of services?
2. Should each service have it's own log or should it all be in one file?
3. Can/Should rustysd just reuse systemd-logind?

Update: I meant systemd-journald of course!

[pwFoo]

Would be nice to have an optional general log file /var/log/rustysd.log and / or logging to syslog... Services should be able to log in it's own file like /var/log/<SERVICE/CUSTOM_NAME>.log? Single file would be ok, but should be prefixed with service name or "system"?

If systemd_logind have more dependencies I would avoid to use / add it...

[pwFoo]

rustysd.log oder .log should be easy to do? Write stderr and stdout to a configured log file. That should be fine for most (simple) setups? Extended logging to (r)syslog or reuse systemd-journald should be optional or not implemented?

[rottrevore]

Personally I recommend the old daemontools approach of every daemon getting its own logger service which handles both writing and rotation at the same time, to avoid TOCTTOU races and to enable greater fault tolerance by decentralizing logging, including easier privilege separation.

JdeBP has written much on this:

http://jdebp.eu./Softwares/nosh/guide/logging.html http://jdebp.uk/FGA/do-not-use-logrotate.html https://news.ycombinator.com/item?id=19084294 https://news.ycombinator.com/item?id=10491489

[KillingSpark]

What's nice about systemd-journald is that you can look at logs of different services that are correlated in time, which the daemontools approach does not provide

[rottrevore]

There's no reason you can't pattern match on a TAI64 timestamp or sort by modification time in a post-processing filter. But I think in general 2) is a false dilemma -- per-service logs can always be coalesced.

[KillingSpark]

I guess that's true. I am currently leaning towards having separate logs, each line tagged with at least a timestamp. But that would require writing a tool that can extract the info you want similar to journalds filtering.

OTOH it could be fun to go the journald route but with a more sensible format. Something like bson which is relatively simple and efficient to scan through.

[KillingSpark]

Also support for the syslog backend for the log crate should be implemented (Hidden behind a feature flag).

[icefoxen]

As a user who interacts with systemd's logging stuff a lot, you're going to need log filtering and correlation tools no matter what, so the best option is to make the logs easy for those tools to handle and easy to use without those tools. Text logs, please god text logs, with optional binary indices if you want them to make filtering and searching fast. Individual logs per service are necessary, because you need to be able to store, rotate and back up those logs individually, while a combined log for everything is convenient but not what I'd choose if I could only have it one way.

Also BSON uses C strings and thus fails my criteria for "sensible", but YMMV.

[pwFoo]

I like small binaries / tools. So logging solution should be simple or if possible optional? (r)syslog would be nice. simple log files fine for me. I don't need journald logging feature.

[pwFoo]

@KillingSpark Would it possible to add log option to unitfile and just write cmd stdout / stderr to the logfile? I don't know if there is such a feature in systemd, but it would be nice for debugging and simplest logging for rustysd I think?

[KillingSpark]

Systemd has an option to redirect stdin/out/err, I could add that. Then you could redirect stdout/err to a file if you wanted to

[pwFoo]

Any plans to add that feature? Would clear my console and redirect all output to "log" files.

[KillingSpark]

Sorry I am currently a bit swamped with university stuff. I do plan to add redirecting everything to one file, for time being until I/we have figured out a good solution.

[pwFoo]

You mean one log file for each unit and one more log file for rustysd itself? With configurable --log_dir (/var/log/)?

[KillingSpark]

@pwFoo Redirecting stdout/err to files is now (ce953310beac3f3ac20304e611386884d3478b7c) supported. Have a look at the test_units/oneshot.service. If you redirect all services to their respective log files the only output on the terminal should be from rustysd.

Next up would be to make rustysd itself write to a file / multiple files (but you as long as it is started by another script, you can easily redirect rustysds output).

[pwFoo]

Works fine for services, thanks!