Add rclone backend

[tamalsaha]

https://restic.net/blog/2018-04-01/rclone-backend

[warmfusion]

I'd like this as the Restic library does not work nicely with our on-prem IBM Cloud Storage (CleverSafe) but we can get the system to work via RClone's implemention of the AWS API. If Stash could use RClone that'd let us bridge backups onto that system nicely.

[tamalsaha]

Should be doable https://restic.net/blog/2018-04-01/rclone-backend

On Tue, Aug 20, 2019, 6:05 AM Toby Jackson notifications@github.com wrote:

I'd like this as the Restic library does not work nicely with our on-prem IBM Cloud Storage (CleverSafe) but we can get the system to work via RClone's implemention of the AWS API. If Stash could use RClone that'd let us bridge backups onto that system nicely.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stashed/stash/issues/408?email_source=notifications&email_token=AAAXEXX64OPZ3KLA437C5WLQFPTY7A5CNFSM4EYLF372YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4WGZIY#issuecomment-523005091, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAXEXVYWYW6UDSKHF4HMU3QFPTY7ANCNFSM4EYLF37Q .

[apvlv]

I also would like to see progress on this. Thank you!

[unixfox]

Restic can now work directly with rclone without any need to use a REST API server: https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone

It's just "as simple" as specifying rclone as a repository backend for restic. The amount of changes into stash should be very minimal due to the great integration between restic and rclone.

[hossainemruz]

Thank you @unixfox for letting us know. We are currently working on Stash v1beta2 API. Once the new API is ready, we can add this feature. I have already added this in the feature queue.

[unixfox]

For anyone looking for using rclone into stash, it is possible with minor modifications. This will restrict the ability to use REST servers, though. Basically, you just have to replace rest: by rclone: in https://github.com/stashed/stash/blob/master/vendor/stash.appscode.dev/apimachinery/pkg/restic/setup.go#L308. Then recompile the Docker image of stash with rclone included in it. You can then modify the default docker image using --set stash-community.operator.registry=yourusername --set stash-community.crdInstaller.registry=stashed when installing stash with helm.

If you don't want to rebuild a Docker image, I maintain these modifications under this Docker image: https://quay.io/repository/unixfox/stash. Use my image with --set stash-community.operator.registry=quay.io/unixfox --set stash-community.crdInstaller.registry=stashed when installing stash with helm.

After that, you can configure your Repository configurations by typing the rclone remote name after url: like:

apiVersion: stash.appscode.com/v1alpha1
kind: Repository
metadata:
  name: rest-repo
  namespace: demo
spec:
  backend:
    rest:
      url: ftp:backup
    storageSecretName: rest-secret

A storage secret name is required before creating a Repository like this, more about that in the docs: https://stash.run/docs/v2021.11.24/guides/latest/backends/rest/

Don't forget to specify how rclone will have to use this remote using the environment variables inside your BackupConfiguration like this example:

apiVersion: stash.appscode.com/v1beta1
kind: BackupConfiguration
metadata:
  name: ss-backup
  namespace: demo
spec:
  repository:
    name: rest-repo
  schedule: "*/5 * * * *"
  target:
    ref:
      apiVersion: apps/v1
      kind: Deployment
      name: stash-demo
    volumeMounts:
    - name: source-data
      mountPath: /source/data
    paths:
    - /source/data
  runtimeSettings:
    container:
      env:
        - name: RCLONE_CONFIG_FTP_TYPE
          value: "ftp"
        - name: RCLONE_CONFIG_FTP_HOST
          value: "1.1.1.1"
        - name: RCLONE_CONFIG_FTP_USER
          value: "username"
        - name: RCLONE_CONFIG_FTP_PASS
          value: "mypass"
  retentionPolicy:
    name: 'keep-last-5'
    keepLast: 5
    prune: true

The environment variable is formed like this: RCLONECONFIGREMOTENAME_PARAMETER. You can find the parameter on the remote type documentation, example for ftp: https://rclone.org/ftp/#standard-options It's not possible to do it using a rclone config file, as you can't specify what files to mount in the stash sidecar container.