search

azlux / log2ram

ramlog like for systemd (Put log into a ram folder)
MIT License
2.62k stars 193 forks source link

journald: Corrupted journal logs due to early backup on shutdown/reboot #185

Closed Kartoffelbauer closed 1 year ago

Kartoffelbauer commented 2 years ago

Hi there, I'm a big fan of log2ram and use it everywhere. However, I've encountered some very annoying problems with journald in conjunction with log2ram.

The Problem

I've noticed that on every shutdown or reboot, journald always marks the last journal logs in /var/log/journal/<machine id>/ as corrupted (with a ~ at the end). This is persistent and there isn't a single archived file without the tilde (~) at the end.

The reason for this seems to be that log2ram creates the backup (/var/log/hdd.log) before the journald.service exits, and therefore the journal log included in the backup is incomplete. So when log2ram loads the backup into memory at the next boot and journald looks for the latest logs, it will determine that they are incomplete and therefore mark them as corrupt (by adding a tilde ~).

This in turn creates a whole new problem, as systemd in its current version on Raspberry Pi OS does not delete these corrupted logs and therefore ignores the setting for SystemMaxUse. This causes the journal folder in /var/log to grow and grow until it fills all the memory allocated by log2ram.

Possible solution

I'm no expert on this, but what actually needs to happen is that log2ram creates the backup after journald.service has finished writing to the journal logs. For the time being, it seems best to simply disable persistent logging by setting the Storage option in /etc/systemd/journald.conf to volatile.

Version info

OS: Raspberry Pi OS bullseye (latest) log2ram: v1.6.1

nodefeet commented 2 years ago

For the time being, it seems best to simply disable persistent logging by setting the Storage option in /etc/systemd/journald.conf to volatile.

Sorry for hijacking your important issue, but wouldn't that basically bypass log2ram?

azlux commented 2 years ago

Hi, Can you try to modify the systemd service with systemctl edit log2ram.service, this create an override file here /etc/systemd/system/log2ram.service.d/override.conf (you can delete this file once you've finish testing.

Can you try adding more service into the Before= instruction ? Especially the systemd-journal-flush.service how is one you're showing making issue in your case.

Kartoffelbauer commented 2 years ago

For the time being, it seems best to simply disable persistent logging by setting the Storage option in /etc/systemd/journald.conf to volatile.

Sorry for hijacking your important issue, but wouldn't that basically bypass log2ram?

Yes, in this case journald will not write anything to /var/log anymore. Instead, everything is logged to /run/log (which is also mounted in RAM) and therefore log2ram is bypassed entirely.

Kartoffelbauer commented 2 years ago

Hi, Can you try to modify the systemd service with systemctl edit log2ram.service, this create an override file here /etc/systemd/system/log2ram.service.d/override.conf (you can delete this file once you've finish testing.

Can you try adding more service into the Before= instruction ? Especially the systemd-journal-flush.service how is one you're showing making issue in your case.

I will try that in the comming days. Thank you very much.

UPDATE: Today I tried your suggestion, but unfortunately nothing changed. Problem remains the same.

solracsf commented 1 year ago

Try this: https://unix.stackexchange.com/a/609594/527701 It may help you analyze where it breaks and how to solve it.

HyP3r- commented 1 year ago

I have created for that a pull request: https://github.com/azlux/log2ram/pull/200

Kartoffelbauer commented 1 year ago

I have created for that a pull request: #200

Thank you very much! journald now works properly with log2ram.