search

elastic / elasticsearch-cloud-aws

AWS Cloud Plugin for Elasticsearch
https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2
577 stars 181 forks source link

Update aws-sdk-java to 1.10.11 #234

Closed lusid closed 8 years ago

lusid commented 8 years ago

Fixes #233

Amazon SDK v1.9.34 S3 functionality used by elasticsearch-cloud-aws no longer works after OpenJDK v1.8u60 due to JodaTime issues with timezone. More information here: https://github.com/aws/aws-sdk-java/issues/444

dadoonet commented 8 years ago

See also https://github.com/elastic/elasticsearch/pull/12859

lusid commented 8 years ago

Any idea how long it will take for a change like that to go through review and possible make it back to ES 1.5 (or 1.7 at least)?

dadoonet commented 8 years ago

For now I'm reopening your PR for this repo in case we plan to fix it for 1.7 and previous versions.

lusid commented 8 years ago

Thanks, I appreciate it. Considering this happened using the official ElasticSearch repository in DockerHub and was sparked by them making a modification to all of their base images as far as I can tell (in this case the java:8 repo), I would imagine there will be many others having this problem soon if they aren't already.

dadoonet commented 8 years ago

Anyone is up to run a test on a test cluster ?

bin/plugin install cloud-aws --url https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-cloud-aws/2.7.1-SNAPSHOT/elasticsearch-cloud-aws-2.7.1-20150824.122206-1.zip

The goal is to run elasticsearch 1.7 with this 2.7.1-SNAPSHOT version using either Java 1.7.0_85 or 1.8u60 (or >)

If you guys tries it, could you report here?

Thanks a lot for your help!

pvanderlinden commented 8 years ago

@dadoonet

I still have the same issue with the 2-7.1-SNAPSHOT cloud-aws plugin, running elasticsearch 1.7.1, Oracle java 1.8u60

[2015-08-25 10:46:30,036][WARN ][snapshots                ] [Anomalito] failed to create snapshot [abcde:sdfsdf]
org.elasticsearch.snapshots.SnapshotCreationException: [abcde:sdfsdf] failed to create snapshot
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:260)
    at org.elasticsearch.snapshots.SnapshotsService.beginSnapshot(SnapshotsService.java:290)
    at org.elasticsearch.snapshots.SnapshotsService.access$600(SnapshotsService.java:92)
    at org.elasticsearch.snapshots.SnapshotsService$1$1.run(SnapshotsService.java:211)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to upload object es-test-backup/snapshot-sdfsdf due to AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 645B595151F0F941)
    at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.upload(DefaultS3OutputStream.java:116)
    at org.elasticsearch.cloud.aws.blobstore.DefaultS3OutputStream.flush(DefaultS3OutputStream.java:90)
    at org.elasticsearch.cloud.aws.blobstore.S3OutputStream.flushBuffer(S3OutputStream.java:71)
    at org.elasticsearch.cloud.aws.blobstore.S3OutputStream.close(S3OutputStream.java:89)
    at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:237)
    ... 6 more
dadoonet commented 8 years ago

@pvanderlinden can you tell me what you have in elasticsearch lib directory and in plugins/cloud-aws directory? Could you run a ls -l in both places?

Thanks!

Also, how did you install your JVM? I mean, download from oracle?

pvanderlinden commented 8 years ago

@dadoonet The JVM is from an ubuntu PPA: https://launchpad.net/~webupd8team/+archive/ubuntu/java which does that automaticly for you.

The ls -l:

lib:
total 29208
-rw-r--r-- 1 root root   167735 Jul 29 09:56 antlr-runtime-3.5.jar
-rw-r--r-- 1 root root   448794 Jul 29 09:56 apache-log4j-extras-1.2.17.jar
-rw-r--r-- 1 root root    47359 Jul 29 09:56 asm-4.1.jar
-rw-r--r-- 1 root root    38090 Jul 29 09:56 asm-commons-4.1.jar
-rw-r--r-- 1 root root 14048559 Jul 29 09:56 elasticsearch-1.7.1.jar
-rw-r--r-- 1 root root  6998724 Jul 29 09:56 groovy-all-2.4.4.jar
-rw-r--r-- 1 root root   914597 Jul 29 09:56 jna-4.1.0.jar
-rw-r--r-- 1 root root   794991 Jul 29 09:56 jts-1.13.jar
-rw-r--r-- 1 root root   489884 Jul 29 09:56 log4j-1.2.17.jar
-rw-r--r-- 1 root root  1699875 Jul 29 09:56 lucene-analyzers-common-4.10.4.jar
-rw-r--r-- 1 root root  2563490 Jul 29 09:56 lucene-core-4.10.4.jar
-rw-r--r-- 1 root root    75491 Jul 29 09:56 lucene-expressions-4.10.4.jar
-rw-r--r-- 1 root root   107784 Jul 29 09:56 lucene-grouping-4.10.4.jar
-rw-r--r-- 1 root root   138282 Jul 29 09:56 lucene-highlighter-4.10.4.jar
-rw-r--r-- 1 root root    64021 Jul 29 09:56 lucene-join-4.10.4.jar
-rw-r--r-- 1 root root    36078 Jul 29 09:56 lucene-memory-4.10.4.jar
-rw-r--r-- 1 root root    97173 Jul 29 09:56 lucene-misc-4.10.4.jar
-rw-r--r-- 1 root root   213029 Jul 29 09:56 lucene-queries-4.10.4.jar
-rw-r--r-- 1 root root   391515 Jul 29 09:56 lucene-queryparser-4.10.4.jar
-rw-r--r-- 1 root root   119449 Jul 29 09:56 lucene-sandbox-4.10.4.jar
-rw-r--r-- 1 root root   126794 Jul 29 09:56 lucene-spatial-4.10.4.jar
-rw-r--r-- 1 root root   179011 Jul 29 09:56 lucene-suggest-4.10.4.jar
drwxr-xr-x 2 root root     4096 Aug 14 14:53 sigar
-rw-r--r-- 1 root root   102177 Jul 29 09:56 spatial4j-0.4.1.jar

plugins/cloud-aws/:
total 6456
-rw-r--r-- 1 root root  518808 Aug 25 14:23 aws-java-sdk-core-1.10.12.jar
-rw-r--r-- 1 root root 2166484 Aug 25 14:23 aws-java-sdk-ec2-1.10.12.jar
-rw-r--r-- 1 root root  258578 Aug 25 14:23 aws-java-sdk-kms-1.10.12.jar
-rw-r--r-- 1 root root  594380 Aug 25 14:23 aws-java-sdk-s3-1.10.12.jar
-rw-r--r-- 1 root root   58160 Aug 25 14:23 commons-codec-1.4.jar
-rw-r--r-- 1 root root   62050 Aug 25 14:23 commons-logging-1.1.3.jar
-rw-r--r-- 1 root root   47425 Aug 25 14:23 elasticsearch-cloud-aws-2.7.1-SNAPSHOT.jar
-rw-r--r-- 1 root root  590533 Aug 25 14:23 httpclient-4.3.5.jar
-rw-r--r-- 1 root root  282269 Aug 25 14:23 httpcore-4.3.2.jar
-rw-r--r-- 1 root root   39815 Aug 25 14:23 jackson-annotations-2.5.0.jar
-rw-r--r-- 1 root root  229998 Aug 25 14:23 jackson-core-2.5.3.jar
-rw-r--r-- 1 root root 1143162 Aug 25 14:23 jackson-databind-2.5.3.jar
-rw-r--r-- 1 root root  589289 Aug 25 14:23 joda-time-2.7.jar
dadoonet commented 8 years ago

@pvanderlinden Everything looks correct to me. I guess that you restarted the node, right?

I heard recently on a similar issue with openJDK 1.7.0.85.

It worked well when using redhead/centos openJDK 1.7.0.85 build 2.6.1.3.el6_6 but not when using openJDK 1.7.0.85 build 2.6.1.3.el6_7

Any chance here you have a similar issue?

pvanderlinden commented 8 years ago

@dadoonet I restarted the node indeed.

Not sure what your question is? What I had before I upgraded: Oracle Java 1.8u45 with cloud-aws on version 2.5.0, that still works fine in an old test instance. On an other test instance I completely removed java, and all downloaded packages for java, reinstalled, but it is still failing with the same error.

dadoonet commented 8 years ago

Did you run at some point ES 1.5 + CloudAWS 2.5.0 but with Latest Java Oracle version on the same machine? Was it working?

I'm trying to understand where the issue is coming from. JVM? Cloud plugin? AWS SDK? ...

pvanderlinden commented 8 years ago

@dadoonet I tried a couple of things on a test machine and other installations Working for me: 1.8u45 with ES 1.5/1.7/1.7.1 and plugin 2.5.0/2.7.1-snapshot Not working for me: 1.8.u60 with ES 1.7/1.7.1 with plugin 2.7.1-snaphot/2.5.0 I did not try ES 1.5 with 1.8u45

dadoonet commented 8 years ago

Sorry you wrote:

Working for me: 1.8u45 with ES 1.5/1.7/1.7.1 and plugin 2.5.0/2.7.1-snapshot

And then

I did not try ES 1.5 with 1.8u45

You meant you did not try ES 1.5 with 1.8u60, right?

Whatever, I saw this comment today: https://github.com/aws/aws-sdk-java/issues/444#issuecomment-134592285

Might be the same concern... Will investigate more.

pvanderlinden commented 8 years ago

I meant that I didn't try ES 1.5 with 1.8u60 indeed.

dadoonet commented 8 years ago

Ok I know what is wrong. I'll update the SNAPSHOT and report again here. Thanks for your help so far!

dadoonet commented 8 years ago

Could you try:

bin/plugin install cloud-aws --url https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-cloud-aws/2.7.1-SNAPSHOT/elasticsearch-cloud-aws-2.7.1-20150826.180934-2.zip

Thanks!

pvanderlinden commented 8 years ago

The new snapshot works fine, create, restore and list snapshot all works again, Thanks.

dadoonet commented 8 years ago

\o/ Thanks again for your help! I really appreciate. Planning to release soonish now.

pvanderlinden commented 8 years ago

Thanks for fixing. This should fix my last problem with the ELK stack hopefully.

peter-svensson commented 8 years ago

Will there be a release 2.7.1 with this fix anytime soon?

dadoonet commented 8 years ago

@pliljenberg Definitely yes! Very very soon!

dadoonet commented 8 years ago

@pliljenberg @pvanderlinden @lusid 2.7.1 and 2.6.1 have been released.

dbaggott commented 8 years ago

@dadoonet, any chance of getting 2.7.1 into the maven repository? As best as I can see, only 2.7.0 is available. Thanks!

dadoonet commented 8 years ago

@dbaggott weird! Sounds like it was partially released. Indeed, artifacts are not in maven repo. Will fix it hopefully tomorrow. Thanks for reporting!

dbaggott commented 8 years ago

@dadoonet, thank you!

dbaggott commented 8 years ago

ps you probably already noted this but all of the fixed versions are missing: 2.4.3, 2.5.2, etc

dadoonet commented 8 years ago

No I didn't. Need to fix that.

dadoonet commented 8 years ago

@dbaggott I looked at it.

I just pushed the last 2 versions. It will need some minutes I guess to be synchronized on maven central.

Thanks again for raising that!

dbaggott commented 8 years ago

Fantastic, thank you. I can confirm that both 2.7.1 and 2.6.1 are now available (despite not showing up in the UI yet).

smiller171 commented 8 years ago

Has this fix been backported to ES 1.5?

dadoonet commented 8 years ago

No. You should update to elasticsearch 1.7 and update AWS plugin accordingly.

dadoonet commented 8 years ago

I meant it has been backported but this branch has not been released since.

EdgeCaseBerg commented 6 years ago

@smiller171 I assume you already figured out a solution, but just in case someone else comes along and is really trying to get something to work. I got the snapshot/repository stuff working by doing the following:

  1. Install 2.5.2 snapshot

    bin/plugin install cloud-aws --url https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-cloud-aws/2.5.2-SNAPSHOT/elasticsearch-cloud-aws-2.5.2-20150822.225952-55.zip

  2. Download some jar files: Joda 2.8.2 AWS SDK Core 1.10.15 AWS SDK Core 1.10.15

  3. mv the old versions of those jars out of the plugins/cloud-aws directory and move those new jars in

  4. Restart ES

It worked for me doing this. But I only have to do this for a local environment running Java 8 I have (since we're running Java 7 where our actual cluster is) so I don't think I'd really recommending downloading a snapshot and changing out jars in the plugin directory. It'd probably be a little safer to replace the other AWS jars as well, but like I said, I didn't because on my local environment I didn't need to test any of the other features of the cloud-aws plugin

Sorry for resurrecting an old thread after the solution for people running 1.5 is to migrate to a different version was given by the maintainer!