search

offen / docker-volume-backup

Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage
https://offen.github.io/docker-volume-backup/
Mozilla Public License 2.0
2.03k stars 86 forks source link

[Feature Request] Support backup targets like Box / Dropbox #103

Open lonix1 opened 2 years ago

lonix1 commented 2 years ago

I'm not a go dev, but maybe someone else knows how to do this... Thanks!

m90 commented 2 years ago

If anyone wants to add this, there seems to be an official (published by Dropbox) "unofficial" Dropbox SDK for Go: https://github.com/dropbox/dropbox-sdk-go-unofficial which can do all we need to do here.

lonix1 commented 2 years ago

Nice find.

For Box: has many dev tools and SDKs but not an official Go SDK; however it does have a CLI which (I assume) could be wrapped in go code.

Box is nice because their free tier is a massive 10GiB vs DropBox's 2GiB. Both are good though.

m90 commented 1 year ago

Dropbox is now supported as of v2.30.0

MaxJa4 commented 1 year ago

Haven't heard of Box before. Is it widely used in some countries or am I the only one who never heard of them before?

On another note, if we include Dropbox, perhaps Google Drive make sense as an option too. There is an official Go SDK for it and it also uses OAuth2 like Dropbox.

lonix1 commented 1 year ago

Haven't heard of Box before. Is it widely used in some countries or am I the only one who never heard of them before?

The latter I'm afraid :-) Box is gigantic and in the same league as Dropbox - and like I said above very common in business environments.

m90 commented 1 year ago

The latter I'm afraid :-) Box is gigantic and in the same league as Dropbox - and like I said above very common in business environments.

Gotta be honest, I also never came across Box outside of this issue.

On another note, if we include Dropbox, perhaps Google Drive make sense as an option too.

Google Drive is probably heavily used, which is good, but I am a bit worried about API stability here as Google is pretty notorious for just changing things when they feel like it. It's probably not possible to connect to Google Drive using some open standard, e.g. WebDAV, SSH, S3? Also: as we've just been bitten by this: is it possible to run a mock Google Drive in tests instead of having to resort to a live service?

lonix1 commented 1 year ago

I also never came across Box outside of this issue.

Info: https://en.wikipedia.org/wiki/Box_(company) NYSE listed.

Just keep in mind you (probably?) won't find larger storage accounts elsewhere in a free tier.

MaxJa4 commented 1 year ago

Google Drive is probably heavily used, which is good, but I am a bit worried about API stability here as Google is pretty notorious for just changing things when they feel like it. It's probably not possible to connect to Google Drive using some open standard, e.g. WebDAV, SSH, S3? Also: as we've just been bitten by this: is it possible to run a mock Google Drive in tests instead of having to resort to a live service?

Didn't find anything about WebDAV etc. but there is also a OpenAPI spec for it, just like with Dropbox: Spec.

Edit: There is also a spec for Box. Edit2: Box works similarly to Dropbox on the API level. Although using the CLI seems complicated (has to be installed via NPM), we could implement it by calling the REST API directly - we just need a few routes anyway. The docs are pretty good and the upload session architecture is similar to Dropbox's.

MaxJa4 commented 1 year ago

Just saw that Box limits the total size of one file to 250 MB for the free tier... so the '10GB for free' is only for marketing purposes, unless you have many small files. For backups, I'd say backups are rarely below 250 MB and also, Box doesn't know anything about paths, it only uses IDs for their API - which doesn't exist for Go, so we'd need to call their REST API directly. Lastly, everything about Box is geared towards companies and business use, which doesn't make it easier to e.g. create a new app inside Box to try things out with test access tokens. All that combined is making it quite complicated and let's me pause on that for now if I'm honest.

lonix1 commented 1 year ago

The 250Mib thing is surprising, maybe our backups are small.

EDIT: oh just checked we create a series of gzip files of ~250 each. :)

Well, thanks for looking into it. After your research which provider do you recommend?

MaxJa4 commented 1 year ago

Well if you don't need a long history of backups, maybe the 2GB of Dropbox is already enough. OneDrive has a 5 GB free tier (not yet implemented here). Google Drive even has 15 GB free tier (not yet implemented here). Backblaze B2 has 10 GB free tier (egress will be free too starting at October). You can use this with the S3 backend.