search

bobafetthotmail / folder2ram

mount those folders to ram without losing access to their counterpart on disk!
GNU General Public License v3.0
110 stars 21 forks source link

journald journals corrupt #32

Closed ryecoaaron closed 3 weeks ago

ryecoaaron commented 1 month ago

When folder2ram rsyncs the files from /var/log/journal, journald is running. On startup, those files are also rsync'd back while journald is running. journald then complains about corrupted journal files and replaces the file causing log entry loss. Since Debian 12 moved from syslog to journald, this can cause a lot of loss if the journals haven't been rotated recently.

It seems like the sync should happen when journald is not running. The question is how to do this. On startup, the folder2ram_startup service could start before journald. On shutdown, the service could shutdown after journald. For a on-demand sync, it could stop journald, sync, and then start journald. I'm sure there are other ways as well.

Thoughts on this?

bobafetthotmail commented 1 month ago

Nothing can really "start before journald", it's started together with systemd in the initramfs before the root filesystems are mounted and is responsible of logging the boot sequence. Then when the root filesystem is mounted it runs "systemd-journal-flush.service" to move the logging to the designated folder on disk.

I'm not sure about the shutdown sequence.

Due to systemd and journald integration, stopping journald isn't just "systemctl stop journald", it will ignore/restart itself immediately.

I think the only way is calling journalctl --rotate before folder2ram's startup, sync, shutdown functions run (this can be done by folder2ram). From the man: "Journal file rotation has the effect that all currently active journal files are marked as archived and renamed, so that they are never written to in future. New (empty) journal files are then created in their place. " https://manpages.debian.org/bookworm/systemd/journalctl.1.en.html

This means that the log files until that point are saved as static files that can be moved safely by rsync. Logging continues in this new journal file, that is very soon lost and replaced with the "corrupted journal file" error when folder2ram is switching the mount points.

So it's not optimal but it should reduce the lost logs to a few minutes

ryecoaaron commented 1 month ago

Thanks for looking into that more. journalctl --rotate sounds like a good improvement.

bobafetthotmail commented 1 month ago

I committed the change for this and also for the other bug report. In my testing it seems ok, I don't see any corrupted files with journalctl --verify