Borg not sending app data

[TILvids]

I'm using the Borg app (via Yunohost) to send to a backup of the PeerTube app to BorgBase server. I've been able to make a connection to BorgBase and it's pulling backups, but they're much smaller than they should be (the full app including video data is over 300GB, the backup says it's taking up about 1.85GB). In the logs, it shows that it's logging in dozens of times, usually sending nothing but occasionally pulling 150-200MB. It does this every night (when I've instructed it to do). I've told the Borg app to backup everything including all apps, but it looks like it's just ignoring the video data directory.

Am I missing something, is there a problem, or is the Borg app not able to back up the data directory for PeerTube?

[orhtej2]

If you create backup via webadmin does the result TAR contain the media files?

[TILvids]

Do you mean a backup via Yunohost? i.e. "Backups >> Local Archive >> New Backup"?

Update I tried running a backup in Yunohost. The size was about 1GB, which is much smaller than what the video data directory should consume. So it looks like maybe the problem is in what Yunohost is backing up, rather than what Borg is doing? Not sure how to get the Yunohost backup to pull the video data directory though (looks like I had all backup options selected around PeerTube).

[orhtej2]

Theoretically PeerTube_ynh backs up data, not sure what the issue is.

Did you change data directory path after installation? Do you have sufficient space left on drive to create the backup?

The obvious question I never asked is if you're running up to date YNH and apps versions?

[TILvids]

Did you change data directory path after installation?

Not to my knowledge.

Do you have sufficient space left on drive to create the backup?

As far as I know (330GB data, 1TB storage)

The obvious question I never asked is if you're running up to date YNH and apps versions?

Yup. I did a YNH update and should be on latest.

Thanks for the ideas!

[orhtej2]

Did you get any error mails on main admin user?

The backup creates a number of log files, including /var/log/backup_borg.err and /var/log/${borg_id}/${current_date}.err (borg_id is placeholder for Borg installation name, typically 'borg', current_date is obviously backup date) accessible via shell. Does any of these include any errors?

[orhtej2]

I installed both peertube and borg locally and I'm unable to reproduce the issue locally. Both 'normal' TAR backups as well as Borg backups created with sudo yunohost backup create --methods borg --apps peertube contain video data, either in apps/peertube/backup/home/yunohost.app/peertube/storage/streaming-playlists or apps/peertube/backup/home/yunohost.app/peertube/storage/videos if the video was not transcoded yet.

[fflorent]

I also installed PeerTube with Yunohost for a while now, with borg to backup.

As far as I can remember, it always backed up everything. I just checked that a recent video was present in the latest backup.

What command do you use to check the backup using borg? borg list, borg export-tar or borg mount?

[TILvids]

Do the backups (either borg or the one in yunohost) accurately reflect the size of the backup? Mine should easily be over 300GB if it's actually pulling the video data into the backup, and like I said, it looks like it's around 2GB in size. This is the same whether it's a yunohost TAR backup or what's on Borg. It leads me to believe yunohost is somehow not accurately pulling backups (so this might not even be a borg issue, because it DOES appear to be pulling a backup similar in size to what gets generated in the yunohost TAR backup).

I wonder if it's a permissions problem or something? I don't know though, the site otherwise runs just fine, it's just these backups that don't appear to be pulling the data in correctly. Thanks for trying to verify!

[fflorent]

Could you run this command:

sudo yunohost backup create -n test -o /tmp  --apps peertube --debug

At the end of the operation, you should see a message like this one:

To view the log of the operation 'Create a backup archive', use the command 'yunohost log show 20230216-070722-backup_create'

Change yunohost log show 2023.... into yunohost log share 2023.... (show → share), this will submit the logs to paste.yunohost.org (redacting the passwords) and share the URL generated here.

This will help us diagnose your issue.

[TILvids]

Here you go @fflorent

https://paste.yunohost.org/raw/ofumoduquy

Thanks!

[fflorent]

2023-02-17 05:34:37,729: DEBUG - ++ ynh_app_setting get peertube do_not_backup_data
2023-02-17 05:34:37,837: DEBUG - + do_not_backup_data=1

Very probably due to that stored setting. Try this command as root:

yunohost app setting peertube do_not_backup_data -v 0

And attempt another backup to see

[TILvids]

@fflorent

After issuing that command, can I do the backup via the web interface, or do I still need to do it via the terminal?

[fflorent]

Whatever you want, web interface is fine :)

[TILvids]

@fflorent

"The archive will contain about 345.1GB of data."

Well, that looks more promising. :slightly_smiling_face: Hopefully everything goes through, thanks for the help!

I wonder why

yunohost app setting peertube do_not_backup_data -v 0

Was set to not back everything up. I don't see any config option (at least in the web interface) that would allow me to change the value, not sure why it'd default to not back all data up.

[TILvids]

Just to wrap this one up, the automated backup ran and instead of 2.84GB, it jumped up to almost 400GB. Looks like everything is working now. Thanks for your help @fflorent @orhtej2 !