Add remote borg backup support

timdiels commented 1 month ago

This adds support for backing up directly to a remote borg repo. (I tried reaching out earlier)

Why?

I'd like to backup/restore via the UI, resist ransomware and store the backups remotely without storing a local (compressed) copy of the nextcloud data.

Alternatives (https://github.com/nextcloud/all-in-one?tab=readme-ov-file#are-remote-borg-backups-supported):

Mount a network file system: Ransomware can encrypt the backups on the network storage.
rclone: Ransomware and stores a local copy of the borg repo
borg backup the local repo to borgbase: Stores a local compressed copy of the nextcloud data as at least 1 borg backup
create your own solution: backups cannot be managed through the UI and others will have to reinvent my work

How to use it

Instead of entering a local repo path (which remains supported), you can choose to instead enter a remote borg repo url. (These screenshots are from a disaster recovery but it's analogous, just ignore the passphrase):

Screenshot from 2024-06-09 15-26-41

The first you try to initialise the repo, the backup container will create an ssh key and foolishly try to init the borg repo resulting in error because you have to authorise the ssh key it generated first. So, e.g. I copy paste the public key shown here to my borgbase repo:

Screenshot from 2024-06-09 15-27-01

Then I can try again as instructed and it should work. The following backup info is shown:

Screenshot from 2024-06-09 14-59-30

Changes

Support for storing backups directly at a remote borg repo (Replace commands to not assume the repo is local, e.g. use BORG_REPO env var, ...)
Exclude files from backups if we don't plan on restoring them anyway, such as the audit log
Restore with borg extract because, at least with a remote repo, borg mount is very slow (20 seconds vs 24 minutes restore for pretty much an empty nextcloud install).

FYI Oddly the original code never restores host-mounts.

Testing

I hacked the code til I could run local deployments of nextcloud-aio without any builds and manually tested:

Entering invalid stuff into forms I changed
Remote borg repo init
Creating and restoring a backup. I created some files in the master container volume; some in an ignored directory to make sure those are not deleted, some in a directory that is part of the backup, after the backup made some changes and made sure that once restored, the ignored files were left alone, new (non-ignored) files were deleted and changed files restored. I can't really open the user UI with my local test setup.
Checking a backup
Setting up a new nextcloud from backup

TODO

I did at one point test local backups, but I should redo that.
If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great
Update the readme, but I first want to hear your thoughts before putting more time in it

szaimen commented 1 month ago

Hi, first of all thank you for your contribution!

I've thought a bit about this the last week and came to the conclusion that I do not want to maintain this since it requires an additional server for me for testing (with a not so easy to reproduce setup) and has the potential of many users needing help getting this to work (and thus only a limited usecase). So I would still only test local backup also in the future.

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

FYI Oddly the original code never restores host-mounts.

Yes, this is expected and documented.

TODO

I did at one point test local backups, but I should redo that.

Sounds good!

If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Yeah, we could create a dev instance from this if we proceed...

Update the readme, but I first want to hear your thoughts before putting more time in it

Yes

timdiels commented 1 month ago

Hi,

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Note to self:

[x] Update readme
[x] Local repo should use borg mount and rsync
[ ] Test local and remote on dev build

szaimen commented 1 month ago

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

Cool, then we can go ahead with this PR. I've invited you to the repo for easier collaboration :)

I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

Great :)

If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Usually it is indeed always via push to github, wait for build and deploy. So probably it is easier for you to keep your debug branch 👍

gregjohnsonsaltaire commented 4 weeks ago

Very much appreciate the PR guys! I'm happy to help test informally ... I have 3 nextcloud-aio sites of 20GB, 40GB & 350GB doing borg backups to Hetzner Storageboxes that I'm prepared to convert to the PR ... Regards

timdiels commented 2 weeks ago

I tried out local backups on my machine and made all the requested changes I believe. Could we get a a dev build going?

szaimen commented 1 week ago

I tried out local backups on my machine and made all the requested changes I believe.

Nice!

Could we get a a dev build going?

Yes, it should be available via nextcloud/all-in-one:develop-4804

timdiels commented 2 days ago

I'm trying it out on my own server (with a VM snapshot before switching to dev). Will I be able to later update this to :latest after it's included in a release or should I revert my snapshot after testing instead?

timdiels commented 2 days ago

Created a backup on my server, I'll enable autobackup/update and try out restoring after some changes later (might be next weekend):

Duration: 1 hours 2 minutes 27.77 seconds
Number of files: 34348
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                9.93 GB              8.89 GB              8.65 GB

(the duration is pretty much optimal considering my 19 Mbps ISP upload speed)

timdiels commented 1 day ago

I rolled back to an old snapshot because apparently the dev build was older than the latest released nextcloud, so it wouldn't start anymore because e.g. elastic can't roll back. I'll try again on the weekend with a tmp vm instead.

msullivan-lc commented 1 day ago

Appreciate all the work you are doing on this @timdiels. Very much looking forward to this addition!

nextcloud / all-in-one