search

restic / rest-server

Rest Server is a high performance HTTP server that implements restic's REST backend API.
BSD 2-Clause "Simplified" License
902 stars 139 forks source link

Fill/sync restic REST server from local repository #150

Open JsBergbau opened 2 years ago

JsBergbau commented 2 years ago

Output of rest-server --version

Not relevant here.

What should rest-server do differently?

There should be a command line utility like awscli sync or rsync. Just copy/store the contents of a local repository to rest-server.

What are you trying to do? What is your use case?

You could have a second backup location. So first backup is to your local NAS. There you run prune and so on for faster speed. Then you sync this data to your rest server for restic. This makes it far easier and more secure than using SSH and rsync to update local content of the rest-server.

Basically this should be quite easy to achieve. Perhaps there is already a thirdparty tool for that, that is even compatible with restic rest-server. There are no needs for exclude files, data changed after and other fancy stuff.

Did rest-server help you today? Did it make you happy in any way?

Restic is a really great backup tool. I've restored data multiple times and it always worked as it should. So I can rely on restic backup.

rawtaz commented 2 years ago

There is no such thing as restic-server. I can only presume you mean rest-server.

To sync/copy a local repository to where your rest-server runs, all you need to do is copy the repository folder to the place where rest-server would read and write it on that server. There is no need for a specialized sync tool for this - just copy the files. If you're looking to keep the local and remote repository in sync, you can use rsync, rclone or similar.

Please close this issue once the above makes sense :)

JsBergbau commented 2 years ago

Thanks, I've fixed restic-server.

What you write perfectly makes sense. That's the only current possible way. This feature request is to make it easier to use rest-server as second backup location. To directly (automatically) copy onto it you need to authenticate via SSH so a potential security risk. However if this means too much work, you also could use minio object storage server instead of rest-server and then you can use awscli or similar tools to sync local repository to your server.

rawtaz commented 2 years ago

That's a good alternative you mention there. What you ask for eventually lands in the good ol' discussion about whether to use one repository that you sync to a second location vs using two repositores that you back up to (which would decrease changes of syncing corruption from the main repository). I personally always use two different repos primarily for that reason. But I get your suggestion. Really all that's needed is something that talks rest-server's REST protocol, there's nothing more to it.

JsBergbau commented 2 years ago

Your solution doing backup twice with two different locations would be the best solution. Especially should there some corruptions on local backend. Because restic never modifies, only creates and deletes, there was the idea for a special tool to sync to rest-server knowing about this part. So when your local blob gets corrupted after first sync the corrupted blob wouldn't be transmitted to remote repository.

As far as I can see a major disadvantage occurs when pruning: Then you have to download data from remote repository (to re-pack blobs) and then re-upload. When pruning your local repository and then syncing you save the download part.

wojas commented 2 years ago

This would indeed be useful.

If a repo is never pruned, it could also provide a way to recover lost blobs.

Perhaps the tool should also change the repo UUID to ensure restic uses an independent local cache. This is something that an rsync will not do.

wojas commented 2 years ago

Perhaps this is a feature that should be added to restic instead, so that repos can be copied between storage backends? I don't know if the restic architecture makes this easy.

DarkSpir commented 5 months ago

I know this issue has been opened for 2.5 years by now. I just want to point out that during that time restic by itself got the ability to copy snapshots from one repository to another ([https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories](see docs)). It's not the same as a sync feature but now one has the necessary tools to do something like that with a bit of script magic. Of course syncing the repository folder with rsync also works but the copy function provides the ability to use two repositories with different encryption passwords.