Can we use s3 option with ceph endpoint?

[marcus-sds]

Summary

I cant find s3 endpoint setup option with aws s3 sdk. Is it possible to add s3 endpoint url with s3 compatible storage?

[jhuckaby]

I've never heard of such a thing before, so I don't know if it is possible. A quick google of the AWS SDK docs turns up this:

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Endpoint.html

But you have to instantiate a class it seems. I have no way of testing this so I can't really commit a change, but you could try to simulate the Endpoint class by passing in an object into the S3 configuration with the name endpoint and all the same properties:

{
    "Storage": {
        "engine": "S3",
        "AWS": {
            "accessKeyId": "YOUR_AMAZON_ACCESS_KEY", 
            "secretAccessKey": "YOUR_AMAZON_SECRET_KEY", 
            "region": "us-west-1",
            "correctClockSkew": true,
            "maxRetries": 5,
            "httpOptions": {
                "connectTimeout": 5000,
                "timeout": 5000
            }
        },
        "S3": {
            "keyPrefix": "cronicle",
            "fileExtensions": true,
            "params": {
                "Bucket": "YOUR_S3_BUCKET_ID"
            },
            "endpoint": {
                "host": "my-s3-server.com:80",
                "hostname": "my-s3-server.com",
                "href": "http://my-s3-server.com",
                "port": 80,
                "protocol": "http"
            }
        }
    }
}

I really have no clue if this would work tho, but good luck!

[ch-liuzhide]

I have problem about ceph too..., aws-sdk can't used for ceph T.T

[jhuckaby]

I'm sorry, I really have no idea what "ceph" even is. I'll leave this issue open, and research it when I have time, but I cannot give an ETA. If you want, you can develop your own engine plugin for the key/value storage system.

[yasin-ahlatci]

I tried using Google Cloud Storage instead of Aws S3 and I can confirm It worked with the configuration below. So ceph or minio should work too.

    "Storage": {
        "transactions": true,
        "trans_auto_recover": true,

        "engine": "S3",
        "AWS": {
            "region": "europe-west1",
            "credentials": {
                "accessKeyId": "access-key-id",
                "secretAccessKey": "secret-access-key"
            }
        },
        "S3": {
            "connectTimeout": 5000,
            "socketTimeout": 5000,
            "maxAttempts": 50,
            "keyPrefix": "",
            "fileExtensions": true,
            "params": {
                "Bucket": "your-bucket-name"
            },
            "cache": {
                "enabled": true,
                "maxItems": 1000,
                "maxBytes": 10485760
            },
                        "endpoint": "https://storage.googleapis.com"
        }
    },

[jhuckaby]

Thank you for this. I'll add this to the docs.