Borg fails when file changes on disk during archive creation

dmaroulidis commented 6 months ago

Describe the bug

Borg_ynh fails to backup an application when files on disk change during archive creation. Probably related to https://github.com/borgbackup/borg/issues/6989. In my situation specifically, as you can see from the backup logs, matrix-synapse adds to homeserver.log during archive creation which causes borg to emit a warning and probably a signal that backup-with-borg receives and stops the process. I tried stopping synapse before starting the borg.service and the backup completes successfully.

Context

Hardware: VM with VPN
YunoHost version: 11.2.12
I have access to my server: Through SSH | through the webadmin | direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no
Using, or trying to install package version/branch: borg 1.2.8~ynh1

Steps to reproduce

Upgrade borg_ynh to the latest version 1.2.8~ynh1 and get notified of the issue by email every morning.

Expected behavior

A solution would be to stop each service before backing it up and restart it after the backup is finished. A better setup would be to take a snapshot of the filesystem and backup the snapshot which would result in zero downtime, but that cannot be accomplished without a filesystem that supports this.

Logs

https://paste.yunohost.org/raw/duzoyalaho

utzer commented 6 months ago

I got the same problem since the late upgrade of borg backup.

2024-06-03 10:17:49,897: DEBUG - + /var/www/borg/venv/bin/borg init -e repokey ssh://user01@server01/./backup
2024-06-03 10:17:50,872: WARNING - A repository already exists at ssh://user01@server01/./backup.
2024-06-03 10:17:50,890: DEBUG - + set -e
2024-06-03 10:17:50,890: DEBUG - + /var/www/borg/venv/bin/borg create ssh://user01@server01/./backup::_auto_synapse-2024-06-03_10:17 ./
2024-06-03 10:17:50,890: DEBUG - + log_with_timestamp
2024-06-03 10:17:50,891: DEBUG - + tee -a /var/log/backup_borg.err
2024-06-03 10:17:50,891: DEBUG - ++ date +%Y-%m-%d_%H:%M:%S
2024-06-03 10:17:50,892: DEBUG - + sed -e 's/^/[2024-06-03_10:17:50] /'
2024-06-03 10:18:09,837: DEBUG - [2024-06-03_10:17:50] apps/synapse/backup/var/log/matrix-synapse/homeserver.log: file changed while we backed it up
2024-06-03 10:18:10,838: ERROR - Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 298, in hook_callback
    hook_return = hook_exec(
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 416, in hook_exec
    raise YunohostError("hook_exec_failed", path=path)
yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app

CodeShakingSheep commented 5 months ago

Same.

utzer commented 5 months ago

Anyone knows if there is some work done on this and if this here it the right place to report, maybe it should be reported via synapase package?

utzer commented 5 months ago

Sorry, it just bugs me that I get a "backup failed" email every two hours now. :-(

utzer commented 5 months ago

is there a possibility to exclude files from the backup, I mean the log is nice to have in some cases, but not the most important as well, so maybe we can simply exclude it for now?

dmaroulidis commented 5 months ago

@utzer I've looked through the backup script of synapse (the file that exports the file list to be included in the backup) but I couldn't figure out how to exclude the log from backups. That's something the maintainers of ynh_synapse could help with, but that would be a patch.

The bigger issue is that this thing can happen with any service that changes its files while the backup is running. That's why I suggested stopping each service before backing it up. I think this can be done in the yunohost backup framework, so that might be something to take to the yunohost repo or at least the forum.

I reverted to my pre-upgrade backup of borg and I've been using the previous version for the last 3 weeks. I'll update when this is fixed. Note that the same situation (a file is changed during backup) can happen with the previous version as well but the backup won't fail if this happens.

Thatoo commented 5 months ago

I face the same issue since I last updated synapse indeed.

I opened an issue on synapse repository.

Thatoo commented 5 months ago

@Josue-T answered on the issue opened here https://github.com/YunoHost-Apps/synapse_ynh/issues/474#issuecomment-2192632966 Hoping it will help.

utzer commented 1 month ago

@fflorent I wonder what the status of the PR is?

I also wonder if it would be a good idea to add a function to the borg app to executed a command before the borg create starts and one after borg create finished? This way it would also be possible to run "systemctl stop synapse.service" before borg create and "systemctl start synapse.service" after borg create finished.

YunoHost-Apps / borg_ynh