Allow per-repository endpoints

[brutasse]

With this patch, the S3 endpoint is no longer a global setting. This allows creating repositories on AWS s3 and others to S3-compatible services, without doing cluster restarts between snapshots.

$ curl -X PUT 'http://localhost:9200/_snapshot/test' -d '{
    "type":"s3",
    "settings":{
        "bucket":"hello",
        "endpoint":"my-s3.example.com",
        "access_key":"access key here",
        "secret_key":"it's a secret"
    }
}'

I couldn't make this work completely on elasticsearch master (snapshot creation API call properly makes s3 requests but in the end does not respond. Might be an issue with my setup). However I have successfully tried the same patch with ES 1.3 and the es-1.3 branch of this repository. Snapshotting and restoring to pithos, our S3 implementation at @exoscale, worked beautifully.

[brutasse]

Ping? Is there anything I can do to help getting this reviewed?

[tlrx]

Thanks @brutasse for this PR!

This looks good, but I'd like to see unit test that creates a repository with a specific endpoint and checks everything is OK.

Also, can you fix the latest merge conflicts? Thanks.

[brutasse]

@tlrx ok, conflicts fixed. I added support for per-repo protocols too (as mentioned in #119) and wrote an integration test.

The integration test passes on its own using @exoscale's storage service but some other integration tests fail and that might be due to my IAM/bucket setup on AWS.

Tested with the following config:

cloud:
  aws:
    access_key: "key"
    secret_key: "secret"
    test:
      write_failures: 0.0

repositories:
  s3:
    bucket: "test-es-cloud-aws-plugin"
    region: "us-east-1"
    private-bucket:
      bucket: "scoped-test-es-cloud-aws-plugin"
      access_key: "key"
      secret_key: "secret"
    remote-bucket:
      bucket: "test-es-cloud-aws-plugin-eu"
      region: "eu-west-1"
    external-bucket:
      bucket: "test-es-cloud-aws-plugin"
      endpoint: "sos.exo.io"
      protocol: "https"
      access_key: "exoscale key"
      secret_key: "exoscale secret"

If you want API access to exoscale's S3-like storage service, simply create an account on https://portal.exoscale.ch/ (if you skip phone verification your account will be disabled…) and go to Account -> API details to retrieve your API keys. Service is in public beta so currently free of charge.

Let me know how it goes!

[tlrx]

LGTM

Thanks for the PR @brutasse . I fixed some minor problems (see d680b94eb748585ed7eebdce1c4184fe61829ad5).

[brutasse]

Awesome, thanks @tlrx! And thanks for fixing my java :)