Dgraph backup doesn't seem to support non amazon s3 compatible buckets

[scroobius-pip]

I've got my kubernetes cluster on Digital Ocean working properly, but now i want to perform a backup to a digital ocean space which is an s3 compatible alternative to amazon's bucket:

curl -XPOST localhost:8080/admin/backup -d "destination=curl -XPOST localhost:8080/admin/backup -d "destination=dgraph-backup.sfo2.digitaloceanspaces.com"

but i get an error : {"errors":[{"message":"Backup failed.","extensions":{"code":"The path \"sfo2.digitaloceanspaces.com/dgraph-backup\" does not exist or it is inaccessible."}}]}

i've set AWS_ACCESS_KEY and AWS_SECRET_KEY accordingly but to no avail.

[martinmr]

Dgraph backup only supports S3 and minio endpoints. We are using the minio client so if that client supports Digital Ocean, then you should be able to use it for Dgraph. Can you try using the minio go client to try to connect to the same bucket?

https://github.com/minio/minio-go

[scroobius-pip]

@martinmr i've been trying for the past days to get minio-go client to compile, i haven't worked with golang before till now. So might take a bit longer

[scroobius-pip]

So i managed to get the minio-go client to work on digital ocean here's how the code looked:

package main

import (
    "log"

    "github.com/minio/minio-go"
)

func main() {
    endpoint := "sfo2.digitaloceanspaces.com"
    accessKeyID := "ACCESS_K"
    secretAccessKey := "ACCESS_S"
    useSSL := false

    // Initialize minio client object.
    minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
    if err != nil {
        log.Fatalln(err)
    }

    if err != nil {
        log.Printf(err.Error())
    }

    buckets, err := minioClient.ListBuckets()
    if len(buckets) > 0 {

        for _, value := range buckets {
            log.Println(value.Name)

        } // log.Printf("Buckets exist")

    }
    if err != nil {
        log.Printf(err.Error())
    }

    // log.Printf("%#v\n", minioClient) // minioClient is now setup
}

So i tried using the same url scheme as with the go client :

curl -XPOST localhost:8080/admin/backup -d "destination=sfo2.digitaloceanspaces.com" To no avail -

{"errors":[{"message":"Backup failed.","extensions":{"code":"The path \"sfo2.digitaloceanspaces.com\" does not exist or it is inaccessible."}}]}

[martinmr]

I edited your comment to properly format the code.

Two things:

1. The destination passed to dgraph for the backup should include the path to the bucket. For example in our tests this ends up being minio://minio1:9001/dgraph-backup. You might try adding s3:// in front of your endpoint but I don't think this is the issue. The minio client should be smart enough to deal with this.
2. It looks like the issue relates to resolving the domain name. Where are you running dgraph? If it's inside some kind of container, could you log into it, and try to ping sfo2.digitaloceanspaces.com. The message makes me suspect that your s3 bucket is not accessible from the environment that is running Dgraph.

[scroobius-pip]

hmm i've tried the first - which doesn't work, the second might be the issue, cause i've got the alpha running in a k8 container, i'll check and get back to you

[scroobius-pip]

I was able to ping sfo2.digitaloceanspaces.com from the container. Also the export is working great with a normal amazon s3 container : curl -XPOST localhost:8080/admin/backup -d "destination=s3://s3.us-east-2.amazonaws.com/dgraph-backup-dgraph"

[martinmr]

umm. Not sure what else we could try. One last thing. Could you run the program you posted above from your k8 container?

If that works fine, I am not sure what else could cause this. The message includes the word inaccessible which could suggest the credentials are wrong but you mentioned already you set the environmental variables properly.

[scroobius-pip]

hmm possibly, i'll try that

[martinmr]

I was going through the binary backups documentation: https://docs.dgraph.io/enterprise-features/#configure-backup

Could you take a look at the "Configure Minio credentials" section. It looks like the minio client is using different environmental variables for minio buckets. Could you set these variables for your container? The minio client might be using these to try to connect to any S3-compatible bucket, including DigitialOcean buckets.

[scroobius-pip]

Okay will do

[martinmr]

Closing this as I've tested backup with both minio and S3 and didn't find any issues. @scroobius-pip if you are still seeing this issue, feel free to re-open this ticket.