Better log/terminal abstractions

[sourcefrog]

Use the standard Rust log crate.

Keeping a log, not on the terminal, could be helpful for debugging.

[WolverinDEV]

Use the standard Rust log crate.

Why did you came up with this specification? What's the argument against using slog for example?

Asking since I'm playing with the idea of implementing a file log. PS: Seems to be a great tool so far, started testing and using privately already.

[sourcefrog]

Doesn't have to be log if something else works better.

Thanks!

[sourcefrog]

My current thoughts would be:

• maybe use tracing for a debug log
• separately, write a change log with one json dict per line, for every entry added/deleted/changed (https://jsonlines.org/)

Maybe I should split this.

In fact, tracing is already integrated and with -D will write to stdout. This seems to work pretty well for a debug log. There could be an option to send debug log to a file.

[WolverinDEV]

The tracing library looks pretty well fitted for the main logging part.

Using a logging library for your second thought seems a bit off. My understanding of a log-file is a file containing events (especially errors) easy readable and understandable for a human.
Like the console output right now.

But logging file changes/new backuped files is a pretty good point. Maybe even store such "update logs" within the backup archive itself in a machine readable way (json?).
This would definitively be another issue thought.

PS print_filenames with some kind of file logging will do it for the first part imo.

[sourcefrog]

Let's take a step back to discuss objectives. I'll also reply here to your comment in https://github.com/sourcefrog/conserve/pull/173#issue-1327625813.

I think there are several different types of output here and it's useful to cleanly separate them:

• Debug logs: normally off; should only need to be turned on by people who are working on Conserve, investigating a bug, or curious.
• UI messages to the user, e.g. that a backup succeeded and how much was written. These could, potentially, be sent through some non-text UI.
• Progress bars.
• "Batch" or bulk output that might be read by some other program, most obviously a list of files from conserve ls, but also backup -v etc.
• stats, as printed at the end of conserve backup: a bit like a message to the user but also conceptually could be saved as json, etc.

I find it useful to have debug output in the terminal, as -D currently does. It would be useful to also have an option to send debug logs to a file or other destination such as systemd. Conceivably people would want to configure debug logging always on.

When any of these are written to the terminal they need to be coordinated so that they aren't scrambled together, particularly given the high levels of concurrency:

• Nutmeg handles erasing/repainting progress bars for output sent through its message facility.
• At the moment progress bars are just turned off when debug output is sent to stdout.
• Progress bars are also turned off for bulk output.

I think to give a nice UI the user-facing messages need to be specifically designed for how they interact with progress bars. And it does seem like user-facing messages are a different thing to info-level debug messages. In particular text UI can show partial lines as output starts and finishes, but that doesn't work so much for logging.

It would, in a sense, be cleaner to split out everything specific to terminal output to a module so that it can use a GUI, web UI, or whatever. That does not mean it should move to src/bin/conserve.rs; this is orthogonal to which binary uses it. Making this separation requires some concept of how to describe a UI independently of how it's drawn. Perhaps this is better deferred until actually adding a non-terminal UI. Possibly this should be viewed as a sum type/enum of various messages that can be displayed either way.

The non-terminal UI that personally I'd most like to see is a non-interactive background job run from something like cron, in which case it writes only to a log. (Obviously you can juss run it from cron today, but it could be refined.) Perhaps this is an argument that messages like "backup succeeded" and the stats should in fact be info-level log messages? In that case perhaps info+ logs should be routed through nutmeg to the terminal... In the noninteractive mode perhaps progress should be indicated by a message logged every minute or so that things are still happening.

[WolverinDEV]

Hey, thanks to your reply.

I agree with your log categories, but except debug messages, all of them depend on the ui/utility conserve is being used in.
For example the terminal progress bar is totally unneeded for a visual ui or a background task as you mentioned.
My aim is to leave error, process and status reporting to the user of lib conserve.
Internal debug messages, which there are currently none (if I'm not mistaken), would be logged via tracing.

I'm already having concurrency, especially the progress bar already in mind (note this comment) but didn't yet fully worked on that issue.
I've already an idea, but need to see how it works out :)

I'm not aiming to rework the terminal output (may add a level/time prefix), nor reworking the terminal UI the user currently has.
To be exact: The goal is to keep it the exact same, except moving all terminal ui related code where it's supposed to be.
"where it's supposed to be" is the key point here (which could be influence by personal preference). I don't think, ui/display related functions belong to the library itself.
I think of the library as an utility to backup files. Such thinking would not include the concept of "how to display something to the user" which belongs into the binary (for me at least).

Differing this task to later, could be an option but I'm already on it so lets go (except it's totally against your will)!
I'm curios what you think about these thoughts. This ofc is just a proposal and 100% discussable.

[sourcefrog]

Are you contemplating adding some other particular UI? Or do you just see this as a generally-useful cleanup?

Just moving all the code to do with terminal UI to one place is OK. From my experience with other programs that have both a library and terminal interface, I think it's preferable to have it in the library (not the src/bin) but optional. Sometimes people want to call the API but have it draw to a terminal. The binary main should be small.

From skimming the PR, it looks like this is mostly changing text output calls to go through trace macros. That's OK, but I'm not so sure it is the right abstraction. A hypothetical GUI that wants to show that the backup completed doesn't want a log line that says the backup completed, it wants some higher level message that can turn into a notification bubble or something.

Regarding bulk output like filenames, I'm not sure they should go through tracing. That might add timestamps or colors, which would not be helpful if the idea is to feed them into grep, etc.

[sourcefrog]

https://github.com/sourcefrog/cargo-mutants/pull/86 integrates nutmeg and tracing in a broadly similar program and can probably be a model.

[sourcefrog]

A while ago (~2020) I had a Report concept that was also trying to split this better, and that's in some ways similar to @WolverinDEV's Monitor idea.

https://github.com/sourcefrog/conserve/blob/057a3d735cc7f5cdf9ca7f99f1c843c0c307aba9/src/report.rs https://github.com/sourcefrog/conserve/blob/057a3d735cc7f5cdf9ca7f99f1c843c0c307aba9/src/report.rs

It did not really work so I pulled it out, so I'd like to not repeat that. I think the problems were:

• it was a bit too thin of an abstraction over the text UI, speaking of particular strings and progress bars
• objects such as the Archive held Arc/Mutex references, which perhaps is a pattern that would work naturally in other languages but didn't go so well in Rust