search

nathanmarz / storm-deploy

One click deploy for Storm clusters on AWS
Other
516 stars 148 forks source link

JDK 1.7.0_51 breaks `deploy-storm` (JClouds/Guice/Guava) #62

Open aaronlevin opened 10 years ago

aaronlevin commented 10 years ago

JDK 1.7.0_u51 breaks lein deploy-storm. The issue resides in jclouds/guice/guava (see: Upgrading JDK from 1.7.0_45 to 1.7.0_51 causes problems when searching for interface implementations). You will get errors that look like:

org.jclouds.rest.config.SyncToAsyncHttpApiProvider<org.jclouds.rest.HttpClient, A> cannot be used as a key; It is not fully specified.

This has been fixed in versions 1.8.0/1.7.1/1.6.2 of jclouds, however it looks like there were some api changes incompatible with storm-deploy's current use of jclouds.

PS - if you are a hitting this bug, an interim workaround is to downgrade your JDK to 1.7.0_u45. This worked for me and seems to have worked for others.

shouhengy-porch commented 10 years ago

Followed weirdcanada's post and adding to his point. Try this to fixed this bug. Basically what it does is to downgrade jdk to jdk 7u45. You need to have jdk-7u45-linux-x64.tar.gz in your ~/ before you do this.

sudo apt-get purge openjdk-\*
sudo mkdir -p /usr/local/java
sudo mv jdk-7u45-linux-x64.tar.gz /usr/local/java/
cd /usr/local/java
sudo chmod a+x jdk-7u45-linux-x64.tar.gz
sudo tar zxvf jdk-7u45-linux-x64.tar.gz
sudo vim /etc/profile
//paste this to the end of the file:
"JAVA_HOME=/usr/local/java/jdk1.7.0_45
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export JAVA_HOME
export PATH"
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk1.7.0_45/bin/java" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk1.7.0_45/bin/javaws" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.7.0_45/bin/javac" 1

sudo update-alternatives --set java /usr/local/java/jdk1.7.0_45/bin/java
sudo update-alternatives --set javaws /usr/local/java/jdk1.7.0_45/bin/javaws
sudo update-alternatives --set javac /usr/local/java/jdk1.7.0_45/bin/javac
sudo chmod 755 /etc/profile
/etc/profile
sudo chmod 644 /etc/profile
sudo reboot

Then you are good to go.

tbatchelli commented 10 years ago

I just pushed an updated version of storm-deploy into https://github.com/nathanmarz/storm-deploy/tree/update-clojure-jclouds

This version uses the latest jclouds 1.7.1 that doesn’t have the Java 1.7.0_51 issue, and in addition will support the newer AWS hardware profiles.

Could you guys give it a try and confirm it works for you? If so, I’ll merge this back to master ASAP.

shouhengy-porch commented 10 years ago

Hi tbatchelli, tested your fix on project.clj, but this error was what I got. Thanks.

$lein deploy-storm --start --name magi --branch master --commit 1bcc169f5096e03a4ae117efc65c0f9bcfa2fa22 >> dummy.txt
Exception in thread "main" java.lang.ClassNotFoundException: org.jclouds.ec2.services.ElasticBlockStoreClient (ebs2.clj:37)
        at clojure.lang.Compiler.analyze(Compiler.java:5205)
        at clojure.lang.Compiler.analyze(Compiler.java:5151)
        at clojure.lang.Compiler$MapExpr.parse(Compiler.java:2498)
        at clojure.lang.Compiler.analyze(Compiler.java:5194)
        at clojure.lang.Compiler.analyze(Compiler.java:5151)
        at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:437)
        at clojure.lang.Compiler.analyzeSeq(Compiler.java:5369)
        at clojure.lang.Compiler.analyze(Compiler.java:5190)
        at clojure.lang.Compiler.analyze(Compiler.java:5151)
        at clojure.lang.Compiler.eval(Compiler.java:5428)
        at clojure.lang.Compiler.load(Compiler.java:5857)
        at clojure.lang.RT.loadResourceScript(RT.java:340)
        at clojure.lang.RT.loadResourceScript(RT.java:331)
        at clojure.lang.RT.load(RT.java:409)
        at clojure.lang.RT.load(RT.java:381)
        at clojure.core$load$fn__4511.invoke(core.clj:4905)
        at clojure.core$load.doInvoke(core.clj:4904)
        at clojure.lang.RestFn.invoke(RestFn.java:409)
        at clojure.core$load_one.invoke(core.clj:4729)
        at clojure.core$load_lib.doInvoke(core.clj:4766)
        at clojure.lang.RestFn.applyTo(RestFn.java:143)
        at clojure.core$apply.invoke(core.clj:542)
        at clojure.core$load_libs.doInvoke(core.clj:4800)
        at clojure.lang.RestFn.applyTo(RestFn.java:138)
        at clojure.core$apply.invoke(core.clj:542)
        at clojure.core$require.doInvoke(core.clj:4869)
        at clojure.lang.RestFn.invoke(RestFn.java:422)
        at backtype.storm.security$eval1298$loading__4410__auto____1299.invoke(security.clj:20)
        at backtype.storm.security$eval1298.invoke(security.clj:20)
        at clojure.lang.Compiler.eval(Compiler.java:5424)
        at clojure.lang.Compiler.eval(Compiler.java:5415)
        at clojure.lang.Compiler.load(Compiler.java:5857)
        at clojure.lang.RT.loadResourceScript(RT.java:340)
        at clojure.lang.RT.loadResourceScript(RT.java:331)
at clojure.lang.RT.load(RT.java:409)
        at clojure.lang.RT.load(RT.java:381)
        at clojure.core$load$fn__4511.invoke(core.clj:4905)
        at clojure.core$load.doInvoke(core.clj:4904)
        at clojure.lang.RestFn.invoke(RestFn.java:409)
        at clojure.core$load_one.invoke(core.clj:4729)
        at clojure.core$load_lib.doInvoke(core.clj:4766)
        at clojure.lang.RestFn.applyTo(RestFn.java:143)
        at clojure.core$apply.invoke(core.clj:542)
        at clojure.core$load_libs.doInvoke(core.clj:4804)
        at clojure.lang.RestFn.applyTo(RestFn.java:138)
        at clojure.core$apply.invoke(core.clj:544)
        at clojure.core$use.doInvoke(core.clj:4880)
        at clojure.lang.RestFn.invoke(RestFn.java:552)
        at backtype.storm.provision$eval20$loading__4410__auto____21.invoke(provision.clj:1)
        at backtype.storm.provision$eval20.invoke(provision.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:5424)
        at clojure.lang.Compiler.eval(Compiler.java:5415)
        at clojure.lang.Compiler.load(Compiler.java:5857)
        at clojure.lang.RT.loadResourceScript(RT.java:340)
        at clojure.lang.RT.loadResourceScript(RT.java:331)
        at clojure.lang.RT.load(RT.java:409)
        at clojure.lang.RT.load(RT.java:381)
        at clojure.core$load$fn__4511.invoke(core.clj:4905)
        at clojure.core$load.doInvoke(core.clj:4904)
        at clojure.lang.RestFn.invoke(RestFn.java:409)
        at clojure.core$load_one.invoke(core.clj:4729)
        at clojure.core$load_lib.doInvoke(core.clj:4766)
        at clojure.lang.RestFn.applyTo(RestFn.java:143)
        at clojure.core$apply.invoke(core.clj:542)
        at clojure.core$load_libs.doInvoke(core.clj:4800)
        at clojure.lang.RestFn.applyTo(RestFn.java:138)
        at clojure.core$apply.invoke(core.clj:542)
        at clojure.core$require.doInvoke(core.clj:4869)
        at clojure.lang.RestFn.invoke(RestFn.java:409)
        at user$eval5$fn__7.invoke(form-init1449835205619147585.clj:1)
        at user$eval5.invoke(form-init1449835205619147585.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:5424)
        at clojure.lang.Compiler.eval(Compiler.java:5415)
        at clojure.lang.Compiler.load(Compiler.java:5857)
        at clojure.lang.Compiler.loadFile(Compiler.java:5820)
        at clojure.main$load_script.invoke(main.clj:221)
        at clojure.main$init_opt.invoke(main.clj:226)
        at clojure.main$initialize.invoke(main.clj:254)
        at clojure.main$null_opt.invoke(main.clj:279)
        at clojure.main$main.doInvoke(main.clj:354)
        at clojure.lang.RestFn.invoke(RestFn.java:422)
        at clojure.lang.Var.invoke(Var.java:369)
        at clojure.lang.AFn.applyToHelper(AFn.java:165)
        at clojure.lang.Var.applyTo(Var.java:482)
        at clojure.main.main(main.java:37)
Caused by: java.lang.ClassNotFoundException: org.jclouds.ec2.services.ElasticBlockStoreClient
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at clojure.lang.DynamicClassLoader.findClass(DynamicClassLoader.java:58)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at clojure.lang.RT.classForName(RT.java:1566)
        at clojure.lang.Compiler.resolveIn(Compiler.java:5658)
        at clojure.lang.Compiler.resolve(Compiler.java:5621)
        at clojure.lang.Compiler.analyzeSymbol(Compiler.java:5584)
        at clojure.lang.Compiler.analyze(Compiler.java:5172)
        ... 84 more
tbatchelli commented 10 years ago

@mephistorockshard , is this with the 'update-clojure-jclouds' branch? The missing library is not referenced in any of the code in this branch...

jpe42 commented 10 years ago

This is my first time trying storm-deploy, and I got this same error on master but it progresses past that point on the 'update-clojure-jclouds' branch and creates all of the instances. However it finally exits with: ...You are not currently on a branch. Please specify which\nbranch you want to merge with. See git-pull(1) for details.\n git pull \nBuild storm failed\nlogout\n", :server "##.###.###.###"}]

I'm not sure if that is normal or not.

My Config: nimbus.image: "us-west-2/ami-6ac2a85a" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-6ac2a85a" #64-bit ubuntu supervisor.hardware: "t1.micro"
zookeeper.count: 1 zookeeper.image: "us-west-2/ami-6ac2a85a" #64-bit ubuntu zookeeper.hardware: "t1.micro"

Command: lein deploy-storm --start --name marketstem --branch 0.9.0.1

jpe42 commented 10 years ago

I tried out lein deploy-storm --attach --name marketstem and it completed ("Attaching Complete") without any errors, but there is nothing at {nimbushost}:8080. Also tried http://{nimbus ip}/ganglia/index.php but I get:

The requested URL /ganglia/index.php was not found on this server. Apache/2.4.7 (Ubuntu) Server at {nimbushost} Port 80

wisemine commented 10 years ago

@tbatchelli also getting the same issue @weirdcanada described. Still planning on merging fixes into master?

abloomston commented 10 years ago

Hi @tbatchelli, I was getting "RestContext[..] cannot be used as a key; It is not fully specified." using OpenJDK 1.7.0_55. Using branch update-clojure-jclouds fixed that problem for me. Thanks!

wl258 commented 10 years ago

Seeing same error with OpenJDK 1.7.0_65 and branch update-clojure-jclouds

acstevens commented 9 years ago

Using 1.7.0_65 and received this error when running lein deploy-storm: org.jclouds.rest.config.SyncToAsyncHttpApiProvider<org.jclouds.rest.HttpClient, A> cannot be used as a key; It is not fully specified.

Switched over the the update-clojure-jclouds branch: git cloan -b update-clojure-jclouds https://github.com/nathanmarz/storm-deploy

Downloaded the dependencies again: lein deps

The error was now gone. Thanks.

tsgautier commented 9 years ago

Using the update-clojure-jclouds branch worked for me. Any plans to merge it to master??

trigun0x2 commented 9 years ago

I also had this problem and the update-clojure-jclouds also fixed it for me.

chao2zhang commented 8 years ago

update-clojure-jclouds works!