leijurv / gb

backup your files properly to google drive and/or s3
GNU Affero General Public License v3.0
36 stars 6 forks source link

Support distributing across multiple targets #13

Closed oldmud0 closed 3 years ago

oldmud0 commented 3 years ago

I like where you are going with this backup solution. Though I know one of the benefits of gdrive is that it gives me a complimentary backup to any NSA data center, I'd be more interested in using the plethora of free, high quality space that the internet already offers - like DNS resolver caches, forum avatars, Discord attachments, Minecraft skins, other people's git repos, and exposed Zookeeper znodes.

Okay, jokes aside, I am wondering if this program can fit my use case. I am a low tier cheapskate who uses Wasabi for storage. Turns out that their data centers are a bit less reliable than I had hoped, but I still think $5/TB is a great deal, so I want to spread my storage across buckets in different regions without incurring significant cost with full 1:1 mirrors. Currently I use Synology Hyper Backup to backup my NAS to S3, which is great, but the format is completely opaque and I'd rather use FOSS tools.

I wonder how feasible it would be to distribute the backup blobs across multiple buckets or targets, and then track the storage location in the database.

Danke.

leijurv commented 3 years ago

Gb stores every blob in every storage that you give it. So if you worry is that wasabi is unreliable, you could create two storages in two different wasabi regions under the same account, then it would backup to both.

However, I want to spread my storage across buckets in different regions without incurring significant cost with full 1:1 mirrors. GB is a full 1:1 mirror, sorry.

With two storages, I don't think you can do better than "RAID1" though. If what you're imagining is something like RAID5 across 2 of 3 storages, then that is a cool idea but I think it would be a lot of work to implement.

RAID1 across 2 of 2 storages is good enough for me and you still get the reliability benefit of "any 1 can fail".

oldmud0 commented 3 years ago

Okay, I'm glad you understand what I'm looking for - something of a file system distributable over various remote sources. Well, I guess I'll keep researching it. Thanks!

leijurv commented 3 years ago

So you do desire something like RAID5?

oldmud0 commented 3 years ago

It doesn't strictly have to be raid5 - it can be some kind of balancing mechanism.