Borg backup failed to backup synapse application

Thatoo commented 4 days ago

Describe the bug

Borg fails when file changes on disk during archive creation and since I last updated synapse, it makes borg backup failing its backup. An issue has been open on the borg backup app repository but they say that the "maintainers of ynh_synapse could help with". https://github.com/YunoHost-Apps/borg_ynh/issues/159

Context

Hardware: Old laptop or computer
YunoHost version: 11.2.16
I have access to my server: Through SSH | through the webadmin | direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no Synapse version : 1.109.0~ynh1

Steps to reproduce

launching a backup using borg backup app.
backup fails
get error similar to :

2024-06-03 10:18:09,837: DEBUG - [2024-06-03_10:17:50] apps/synapse/backup/var/log/matrix-synapse/homeserver.log: file changed while we backed it up
2024-06-03 10:18:10,838: ERROR - Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 298, in hook_callback
    hook_return = hook_exec(
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 416, in hook_exec
    raise YunohostError("hook_exec_failed", path=path)
yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app

Expected behavior

Borg backup should be successful as it use to be before one of last synapse update.

Josue-T commented 2 days ago

Hello,

So to me there 2 issue linked to this thing.

Fist of all, you ran the backup while the service was running and it's really not recommended, mainly for data integrity reason. When you run the backup this way some data could change while the backup was made and so you will end with some data which depends on other which changed while the backup was running. Note that I's a known issue and there are already some issue about this somewhere else.

Over this previous described issue, we can have the question of should we backup the log in the apps. Until now on the apps that I maintained, the logs was I saved and restored (on backup/restore). But since https://github.com/YunoHost/package_linter/pull/154, we can suspect that the strategy should change.

To me the main issue about this is that normally we should recommend a way to manage the log for all apps the same way. So to me there are 2 possibility:

it must not be saved, restored and removed and everything is managed by the core
the app must manage completely the log with backup/restore install/remove script

Until https://github.com/YunoHost/package_linter/pull/154 to me it was clearly the second one. But since this we have an unclear situation where we request that package don't remove the log but we still need to backup/restore following the example cf https://github.com/YunoHost/example_ynh/blob/master/scripts/backup#L64 which is a bit inconsistent.

Following this situation to me, firstly we must clarify how the log should be managed by all packages and following the decision of this, I'll adapt the package following the packaging guideline.

dmaroulidis commented 3 hours ago

Hi @Josue-T,

Thanks for informing us about the log situation upstream. Until this is implemented, we could stop the service before backup following the advice the backup mechanism gives. The issue here is that Borg backup doesn't do that automatically and I don't see a way that this can be done automatically by the yunohost backup infrastructure.

How can we solve that? I think this needs to happen independently of how we manage logs, for the data integrity reasons you provided.

YunoHost-Apps / synapse_ynh