dadoonet
commented
7 years ago Can't you define cloud.aws.s3.endpoint in such a case?
See https://github.com/elastic/elasticsearch-cloud-aws/tree/v2.7.1/#using-other-s3-endpoint
Closed smjgithub closed 7 years ago
dadoonet
commented
7 years ago Can't you define cloud.aws.s3.endpoint in such a case?
See https://github.com/elastic/elasticsearch-cloud-aws/tree/v2.7.1/#using-other-s3-endpoint
smjgithub
commented
7 years ago Yes, I am defining the S3 end-point but that is an internal DNS address which responds with a redirect.
To allow the owner of that end-point to point somewhere else in the event of system outages I guess only way is to implement some sort of proxy between the AWS Cloud plugin and the S3 end-point so handle the redirect?
If I use the IP of the S3 service directly then it works but they'd prefer me not to hard-code it in this way. The proxy is just my idea to work around this issue?
Ideally it would be good if the AWS Cloud plugin could respond to the 302 but it seems to just treat this as an error?
dadoonet
commented
7 years ago Ideally it would be good if the AWS Cloud plugin could respond to the 302 but it seems to just treat this as an error?
Yes. Actually we are using AWS SDK here which apparently does not support that. Unless there is a magical knob I did not see :)
Thinking out loud: may be this is something which can be reported in AWS SDK project instead?
smjgithub
commented
7 years ago Yes, that seems to make sense! Is that something that you could raise there citing this ticket as a 'use case' calling for it?
smjgithub
commented
7 years ago I'm not familiar with the AWS SDK and only use the plugin so perhaps this additional feature request would be best explained by a developer?
dadoonet
commented
7 years ago BTW do you have in logs the full stack trace?
smjgithub
commented
7 years ago Is this what you're after?
1) Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Found (Service: Amazon S3; Status Code: 302; Error Code: 302 Found; Request ID: null), S3 Extended Request ID: null
at org.elasticsearch.repositories.s3.S3Repository.
1 error
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:360)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:154)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
... 11 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Found (Service: Amazon S3; Status Code: 302; Error Code: 302 Found; Request ID: null), S3 Extended Request ID: null
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3654)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1020)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:978)
at org.elasticsearch.cloud.aws.blobstore.S3BlobStore.
dadoonet
commented
7 years ago I opened https://github.com/aws/aws-sdk-java/issues/903. Will see what is the outcome of this. For now, you need to define the right endpoint.
Please note that this repo is now deprecated and will only get bug fixes. New features will only go to elasticsearch repo.
smjgithub
commented
7 years ago OK, great! thanks for following up on this...
I spoke to the internal provider of our S3 repo and he said that others are using the AWS SDK and they have worked around this 302 redirect problem so I will see if I can find any more info and pass it on if I do...
dadoonet
commented
7 years ago A guess is that we check manually this URL and if we get a redirect we change the configuration you defined with the new URL. But IMO this is something that should be supported by the SDK.
dadoonet
commented
7 years ago So AWS team rejected the feature request for the reasons they exposed there: https://github.com/aws/aws-sdk-java/issues/903
I'm not sure if we want to implement such a thing on our side and I believe we don't want to make the code more complex.
@tlrx WDYT?
smjgithub
commented
7 years ago Fair enough, I've got the end-point hard-coded to the S3 service IP at the moment and working fine.
I guess the proxy approach is still an option if we choose to go down that route.
Thanks for your persistence in following this up :)
tlrx
commented
7 years ago If I understand correctly, the IBM CleverSafe S3 implementation sends redirections to the client but the AWS SDK team indicates that it is not part of the service contract in AWS to send redirect responses?
If that's correct then I don't think we should implement such a thing on plugin's side.
dadoonet
commented
7 years ago Thanks Tanguy. We are on the same page. Closing
smjgithub
commented
7 years ago Thanks (Tan)guys! Best to stick to AWS contract...
AWS Cloud plugin does not seem to correctly handle a 302 Redirect when encountered on the S3 end-point
Not sure if this is a bug or that perhaps we should access the end-point via a proxy of some sort?
Below is the error from the ES log when attempting to create a new S3 repo...
The S3 repo service is actually provided by IBM CleverSafe and should be fully S3 compliant.
Accessing the service directly via IP address works fine where the redirect does not happen but when using via a generic DNS address which redirects then problem occurs...
[2016-10-25 12:05:16,324][WARN ][rest.suppressed ] path: /_snapshot/s4-es-mon-prod, params: {repository=s4-es-mon-prod} RepositoryException[[s4-es-mon-prod] failed to create repository]; nested: CreationException[Guice creation errors:
1) Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Found (Service: Amazon S3; Status Code: 302; Error Code: 302 Found; Request ID: null), S3 Extended Request ID: null at org.elasticsearch.repositories.s3.S3Repository.(Unknown Source)
while locating org.elasticsearch.repositories.s3.S3Repository
while locating org.elasticsearch.repositories.Repository