add other backup target options (sftp, rsync, dropbox)

[JoshData]

Consolidating the issues:

• sftp (closes #553)
• rsync (closes #676, PR in #678)
• dropbox (closes #711)
• B2 (BackBlaze Cloud Storage) (see #836)
• Minio (see #1148)
• Wasabi (S3-compatible) (see #1234)

[ghost]

I would like to see the dropbox backup option. As long as the data is encrypted I don't care where I save it and dropbox would be a lot easier than amazon s3 for most users.

[tkorves]

What about adding Amazon Cloud Drive as target. It's much cheaper as S3. And also more cost effective.

[zoof]

Amazon Cloud Drive seems to be quite slow. Independent of MIAB, I decided to try duplicity with acd_cli to back up my NAS -- I managed to back up about 40G in 24 hours. My NAS is ARM based but neither the CPU nor the RAM was at any point maxed out. I've discontinued my free trial.

[semente]

well, rsync is not working. It just say "Connection to rsync host failed". From command line I can connect to host. Is there a log file with debug info? Thanks

[yodax]

well, rsync is not working. It just say "Connection to rsync host failed". From command line I can connect to host. Is there a log file with debug info? Thanks

I know this isn't very helpful, but it works for me. You can try running the generated duplicity command yourself. Or the backup from the command line. Not sure if there is a log file.

[semente]

@yodax I will. Thanks. But just to make clear, I get this from the control panel, while trying to save my rsync settings it do not allow me to do that and show that message. I will look at the code how this test is performed and I may can get an idea about what is going on.

[semente]

@yodax found the issue, it is in the path:

1) the path must exist on destination 2) I can't use relative paths, looks like if it do not start with / it do not works. I tried with ./ too. It must be considered valid paths 3) Feedback message could include the rsync command output that failed, its error messages are helpful

I guess I should open a new issue to track this, right?

[yodax]

Ah yes, relative paths won't work. A new issue would be great! 👍

[LVLAaron]

Any additional thoughts about Backblaze B2? It's become a very popular object store that many mainstream applications now have integrations with. Would be great to have OwnCloud on the list!

[nomandera]

Putting this here for reference/info.

Since Crashplan dropped all their non commercial customers I have moved to Duplicati running via docker for convenience.

Without trying to "sell" the solution I can testify it just works and is trivial to use and support. It also supports more backends OOTB than we could ever hope to.

Just perhaps this is LGPL tools ticks all the boxes and saves wasting effort.

[binarykitchen]

Totally would welcome support for Backblaze! Any plans on this?

[barrybingo]

I have a fork https://github.com/barrybingo/mailinabox/tree/other-duplicity-backends that allows using any duplicity backend for backups.

The main functionality resides in management/duplicity_target_ls.py This is a standalone python 2.7 script that, given environmental variables and a duplicity target_url, will list the backup archive files. It works by using duplicity's own python modules.

Listings of the the archive files are used within mailinabox as a means to verify backup settings and to show the status of the backup.

My motivation was to use google drive as a backup target as it gives 15GB free which is plenty for me.

To achieve this I had to install pydrive to the system python 2.7 installation via a simple pip install pydrive Other backends might require their own helper modules

custom.yaml is altered to allow environmental variables and ends up looking like this:

envs:
  GOOGLE_DRIVE_SETTINGS: /home/user-data/pydrive-settings.yaml
min_age_in_days: 3
target: gdocs://some.user@gmail.com/duplicity
target_pass: ''
target_user: ''

The target can be any valid duplicity target_url All envs in custom.yaml are applied to the duplicity process

For my gdocs usage I had to create a file /home/user-data/pydrive-settings.yaml

client_config_backend: settings
client_config:
    client_id: blahblahblahblah.apps.googleusercontent.com
    client_secret: thisisasecret

save_credentials: True
save_credentials_backend: file
save_credentials_file: /home/user-data/pycredentials.json

get_refresh_token: True

oauth_scope:
    - https://www.googleapis.com/auth/drive
    - https://www.googleapis.com/auth/drive.install

This in turn uses a credentials file /home/user-data/pycredentials.json which is itself a bit fiddly to create.

The admin web interface has been updated to include editing of environmental vars and look like this

This method of is not without faults and is for advanced users only so I've not submitted a pull request. Maybe it can be cleaned up to be made more user friendly.

At the moment this working for me so I thought I'd share it.

[inglor]

Diplicity supports B2 1 so it should be straight forward to implement this.

[pandruszkow]

Another vote for Backblaze from me. B2 and Wasabi seem to have the best value for money at the moment.

[tentious]

Another vote for Backblaze from me. B2 and Wasabi seem to have the best value for money at the moment.

+1 for Wasabi from me - But really, any S3 compatible storage should be a valid option. If you wanted to roll your own, see Minio.

[just4t]

@tentious See Backup Amazon S3: Added support for custom endpoints and proposed pull request but still waiting to be accepted / merged.

[durd]

I wouldn't mind seeing Minio support, it is an s3-backend but miab is running an old and unsupported boto package and therefore I guess an old version of s3.

There is an embryo using boto3 here and I had a go at understanding the issue here. Being a scriptkiddie I couldn't edit miab properly and was just presented with my url in a dialog box when I saved :)

From what I gather the existing code for actually backing up can be re-used, it's "just" listing the files that is the issue. Scriptkiddie me might have misunderstood.

[ghost]

@durd someone has documented an unsupported hack to get minio working here:

https://github.com/mail-in-a-box/mailinabox/issues/1148#issuecomment-731762792

[durd]

@acuteaura I found and applied that hack soon after my comment. Adding minio as a supported backup target is still an enhancement I'd like to see.