Support for Copying snapshots between repositories

[inode64]

I would like to submit a feature request that I believe could greatly enhance the usability and reliability of your system. Specifically, I would like to request the addition of support for copying between repositories, which would enable users to create remote backups and synchronize repositories as if performing a scheduled backup.

Key Benefits:

• Remote Backup: This feature would allow users to maintain remote backups of critical repositories, ensuring data safety and disaster recovery.
• Scheduled Synchronization: By enabling synchronization between repositories, users could automate backup processes, making them more reliable and less prone to human error.
• Redundancy and Security: Copying between repositories would increase redundancy, adding an extra layer of protection for important projects.

Suggested Implementation:

• Enable copying between two different repositories, either in the same system or across different instances. Integrate a scheduling feature so that synchronization can occur automatically at predefined intervals, similar to a regular backup.
• Provide options for selective copying or full repository replication, to give users control over the data being synchronized.
• I believe this feature would be extremely valuable for teams looking to ensure the security and availability of their repositories. Thank you for considering this suggestion. Please let me know if you need any further information or clarification.

[garethgeorge]

Hey, thanks for the FR. This is something I've seen a number of people surface as a want -- and I think it's reasonable.

So far I've been hesitant though, primarily due to this warning in the restic docs: https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories .

There are some significant footguns here. It may still be reasonable to try supporting with a warning. I think the way to go about it would be through the hook subsystem e.g. as a "CONDITION_SNAPSHOT_SUCCESS" hook that runs a restic copy command to a repo that's defined.

Edit: another piece of complexity i'm seeing is that the restic copy command uses a different set of env vars to specify options -- this one could actually be a blocker as backrest today doesn't try to interpret flags / env vars (just forwards them). Some thought is needed re: how we'd model this.

Definitely open to ideas if there's a low cost way to implement this.

[Janhouse]

As a workaround, for now I am testing the use of repository CONDITION_CHECK_SUCCESS hook to trigger rclone sync command daily.

Using the hook ensures that while the clone is running, it won't do other actions.

Though I would prefer to have some other way to trigger this, so at the end of it I could also trigger the Discord hook and notify of the status.

Here is my config

I have custom `Dockerfile` that adds rclone and config for it ```Dockerfile FROM garethgeorge/backrest:latest-alpine RUN apk --no-cache add docker docker-compose rclone RUN mkdir -p /root/.config/rclone/ COPY rclone.conf /root/.config/rclone/ ``` The `rclone.conf` ```ini [backup-server] type = sftp host = 10.10.10.10 user = backup key_file = /root/privkey disable_hashcheck = true ``` And then in `docker-compose.yml` I mount the private key `- ./id_rsa:/root/privkey` The hook command is basically ```sh rclone sync /backups/restic-repo/ backup-server:backups/restic-repo ```

[garethgeorge]

+1, I think using rclone or rsync type tools in a hook is a good way to do this, it's safe to rely on the fact that hooks block any other operation from running on the repo.

[Janhouse]

I think the downside is that if rclone fails, you have a broken mirror. This could be handled by mirroring to a different folder and then swapping them, but that requires double disk space and existing files would always be uploaded.

I will try using the hooks and do the restic copy, but that can have some issues with file deduplication, as I read in the readme. :face_with_diagonal_mouth:

My big wish is still that backrest would allow selecting multiple restic repos for the same backup plan, without manually having to copy over json configs. :pray:

[garethgeorge]

The question in my mind with supporting multiple repos in the UI is what the expectations around hooks and scheduling should be.

It introduces something of a cross product e.g. plans cross repos in terms of the number of scheduled tasks. It potentially introduces some footguns as well re: how repo hooks and plan hooks might interact across those combinations that need thinking through.

I wonder if some other concept e.g. a path template or similar is the way to go such that large pieces of configuration can be kept in sync between multiple plans? I.e. flags or excludes, etc.

[mattdale77]

+1 for some way to do this. With bare restic I have actually been running the same backup twice, once for each repository. I don't remember exactly why but it was probably to do with issues with copying snapshots. In my case the ability to add a second repository into the plan would satisfy my requirements I think

[redoranges]

I'd like to get this working with command hooks as you suggested, however I'm stuck trying to locate the restic binary to run the command. I tried "restic" and "$BACKREST_RESTIC_COMMAND"

This is my untested first attempt: $BACKREST_RESTIC_COMMAND copy --from-repo /config/from_repository_file --from-password-file /config/from_password_file --repository-file /config/repository_file --password-file /config/password_file

It looks like restic is maybe in a version specific directory?