search

elastic / elasticsearch-cloud-aws

AWS Cloud Plugin for Elasticsearch
https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2
577 stars 181 forks source link

cloud-aws plugin does not put snapshot to s3-ceph API #255

Closed jindov closed 7 years ago

jindov commented 8 years ago

This is my config in elasticsearch.yml for connecting to aws:

cloud:
    aws:
        access_key: my-key
        secret_key: my-secret-key
        protocol: http
        s3:
            endpoint: "http://s3.mydomain.net"
            bucket: "esrepo"
            signer: "S3SignerType"
        signer: "S3SignerType"
repositories:
    s3:
        endpoint: "http://s3.mydomain.net"
        bucket: "esrepo"
        access_key: my-key
        secret_key: my-secret-key
        protocol: http

When I start elasticsearch with enabling debug log, it notice:

[2015-12-03 09:38:34,117][DEBUG][cloud.aws                ] [Alkhema] using AWS AP I signer [S3SignerType]
[2015-12-03 09:38:34,122][WARN ][cloud.aws                ] [S3SignerType] may not  be supported

However, with this signer, I can connect to s3-ceph and verify snapshot repo with this command:

curl -XPUT 'http://localhost:9200/_snapshot/mysnap?verify=false' -d '{"type":"s3","settings":{"access_key":"my-key","secret_key":"my-secret-key","bucket":"esrepo","endpoint":"s3.mydomain.net","protocol":"http"}}'

elasticsearch log
[2015-12-03 09:41:02,589][DEBUG][repositories.s3          ] [Alkhema] using bucket [esrepo], region [null], endpoint [s3.mydomain.net], protocol [http], chunk_size [100mb], server_side_encryption [false], buffer_size [null], max_retries [3], cannedACL [null]
[2015-12-03 09:41:02,590][DEBUG][com.amazonaws.request    ] Sending Request: HEAD http://esrepo.s3.mydomain.net / Headers: (User-Agent: aws-sdk-java/1.10.33 Linux/2.6.32-573.3.1.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.65-b01/1.8.0_65, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
[2015-12-03 09:41:02,590][DEBUG][com.amazonaws.services.s3.internal.S3Signer] Calculated string to sign:
"HEAD

application/x-www-form-urlencoded; charset=utf-8
Thu, 03 Dec 2015 02:41:02 GMT
/esrepo/"
[2015-12-03 09:41:02,593][DEBUG][com.amazonaws.http.impl.client.SdkHttpClient] Attempt 1 to execute request
[2015-12-03 09:41:02,604][DEBUG][com.amazonaws.requestId  ] x-amzn-RequestId: not available
[2015-12-03 09:41:02,604][DEBUG][com.amazonaws.request    ] Received successful response: 200, AWS Request ID: null
[2015-12-03 09:41:02,604][DEBUG][com.amazonaws.requestId  ] AWS Request ID: not available
[2015-12-03 09:41:02,605][INFO ][repositories             ] [Alkhema] update repository [mysnap]
--------------CEPH access -------------------------------
192.168.1.100 - - [03/Dec/2015:09:42:23 +0700] "HEAD / HTTP/1.1" 200 242 "-" "aws-sdk-java/1.10.33 Linux/2.6.32-573.3.1.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.65-b01/1.8.0_65"
-----------------------------------------------------------------------------------------------------------------------

It's connected okey but I can't put anything to repo, it time out and hang on: curl -XPUT 'http://localhost:9200/_snapshot/mysnap/snapshot1?wait_for_completion=true'

--------------------------log------------------------------
[2015-12-03 10:48:05,907][DEBUG][com.amazonaws.request    ] Sending Request: PUT http://esrepo.s3.mydomain.net /meta-snapshot1.dat Headers: (x-amz-acl: private, User-Agent: aws-sdk-java/1.10.33 Linux/2.6.32-573.3.1.el6.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.65-b01/1.8.0_65, Content-Length: 110, Content-Type: application/octet-stream, )
[2015-12-03 10:48:05,907][DEBUG][com.amazonaws.http.AmazonHttpClient] Retriable error detected, will retry in 600ms, attempt number: 1
[2015-12-03 10:48:06,508][DEBUG][com.amazonaws.services.s3.internal.S3Signer] Calculated string to sign:
"PUT

application/octet-stream
Thu, 03 Dec 2015 03:48:06 GMT
x-amz-acl:private
/esrepo/meta-snapshot1.dat"
[2015-12-03 09:58:33,509][DEBUG][com.amazonaws.http.impl.client.SdkHttpClient] Attempt 1 to execute request
[2015-12-03 09:59:26,564][DEBUG][com.amazonaws.http.impl.client.SdkHttpClient] Closing the connection.
[2015-12-03 09:59:26,566][INFO ][com.amazonaws.http.AmazonHttpClient] Unable to execute HTTP request: Read timed out
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
        at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
        at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:769)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382)
        at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.doUpload(DefaultS3OutputStream.java:143)
        at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.upload(DefaultS3OutputStream.java:108)
        at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.flush(DefaultS3OutputStream.java:90)
        at org.elasticsearch.cloud.aws.blobstore.S3OutputStream.flushBuffer(S3OutputStream.java:71)
        at org.elasticsearch.cloud.aws.blobstore.S3OutputStream.close(S3OutputStream.java:89)
        at org.elasticsearch.common.blobstore.support.AbstractLegacyBlobContainer.writeBlob(AbstractLegacyBlobContainer.java:76)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:644)
        at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:211)
        at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:436)
        at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:421)
        at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:58)
        at org.elasticsearch.cluster.service.InternalClusterService$AckCountDownListener.onNodeAck(InternalClusterService.java:772)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:521)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

And when I restart elasticsearch with changing in API type from S3SignerType to AWS3SignerType

---------------eslasticsearch log when restart---------------------------

[2015-12-03 09:54:16,868][DEBUG][com.amazonaws.request    ] Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: null; Status Code: 403; Error Code: 403 Forbidden; Request ID: null), S3 Extended Request ID: null
-----------------------------------------------------------------

I use 1 key for 2 case but AWS3SignerType cannot to access to S3-ceph API so I cannot do anything

Please help me, if u need to more log, inform me

I run this system with es 2.1, latest cloud-aws plugin, jdk 8 (java version "1.8.0_65")

jindov commented 8 years ago

Update output when run verify repo with S3SignerType API:

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[esrepo] path  is not accessible on master node"}],"type":"repository_verification_exception","reason":"[esrepo] path  is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object tests-mVhkDiISRfuIJ66t-pwM3Q/master.dat-temp","caused_by":{"type":"amazon_client_exception","reason":"Unable to execute HTTP request: Read timed out","caused_by":{"type":"socket_timeout_exception","reason":"Read timed out"}}}},"status":500}
awislowski commented 8 years ago

My PR solves this issue with ceph.

rjernst commented 8 years ago

See my comment on the PR. This PR last year should have fixed the issue: elastic/elasticsearch#13360. However, that is marked as 2.1, but I'm not sure it was actually backported. I can't find it anywhere in 2.x. @xuzha Can you check this?

rjernst commented 8 years ago

Oh actually it looks like there was a backport, but it was not exactly the same as master. I think the warning you saw is a red herring, because it is trying to register an already registered type (on master it skips S3SignerType because it has statically registered it).

xuzha commented 8 years ago

In 2.x is slightly different. Taking a look, but I think the S3SignerType is the right type to use for ceph .

rjernst commented 8 years ago

I agree it is the right type. But with the 2.x change, it looks like it will still try to register S3SignerType even though it is always registered (which gets the IAE and prints the warning).

xuzha commented 8 years ago

See here, https://github.com/elastic/elasticsearch/blob/2.1/plugins/cloud-aws/src/main/java/org/elasticsearch/cloud/aws/AwsSigner.java#L53-L60

The warning is actually don't stop user to setup the S3SignerType, we just overwrite the Signer by calling configuration.setSignerOverride(signer). It is not register it, right ?

awislowski commented 8 years ago

true, aws sdk java 1.10.33 (which is used now) is registering it.

jindov commented 8 years ago

As you explain above, I cannot use s3 ceph to store my snapshot with this s3 repository plugin? I've upgraded to the latest version of elasticsearch, 2.2

michalzubkowicz commented 7 years ago

Same question here. Anyone have backup to cloud working with S3SignerType with ES 2.2.1 or 2.3.5? In both I have "[S3SignerType] may not be supported" and amazonaws.services.s3.model.AmazonS3Exception: Gateway Time-out (Service: Amazon S3; Status Code: 504; Error Code: 504 Gateway Time-out; Request ID: null), S3 Extended Request ID: null

jogura-zz commented 7 years ago

I'm currently using ES 2.4.1 with cloud-aws plugin version:

_cat/plugins?v

name     component     version     type
node-1   cloud-aws     2.4.1       j

After some digging it seems that the correct signer is: S3SignerType (at least with this current configuration)

I'm able create the repository but it always fails verification with the following error:

{
   "error": {
      "root_cause": [
         {
            "type": "repository_verification_exception",
            "reason": "[ceph_repo1] path [es-snapshots] is not accessible on master node"
         }
      ],
      "type": "repository_verification_exception",
      "reason": "[ceph_repo1] path [es-snapshots] is not accessible on master node",
      "caused_by": {
         "type": "i_o_exception",
         "reason": "Unable to upload object es-snapshots/tests-ZG71SQBeTWmAgce_0GNF8Q/master.dat-temp",
         "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "null (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: tx00000000000000006b8fe-00583f3b24-6c2f8-default)"
         }
      }
   },
   "status": 500
}

The bucket exist check succeeds with response 200 however the PUT fails with a 403.

Is this a plugin issue or is related to permissions?

I'm able to create directories/buckets using s3cmd and the same access_key and secret_key configured in the elasticsearch.yml.

Does anyone have any ideas?

UPDATE: The problem was not in the plugin and was in the ACCESS_KEY/SECRET and END_POINT. Once the setup was pointing to the correct CEPH instance everything just worked.

dadoonet commented 7 years ago

As @jogura confirmed (and thanks a lot for the update), it seems that CEPH can be used with S3 plugin. So I'm closing this issue.