search

nathanmarz / storm-deploy

One click deploy for Storm clusters on AWS
Other
516 stars 148 forks source link

storm-deploy for "new generation" ec2 hardware #57

Open vanjos opened 10 years ago

vanjos commented 10 years ago

I'm looking to deploy storm (and zookeeper) into our VPC with the new i2 and c3 hardware that has recently come available to see if it makes sense for our setup.

I know there may be some issues with VPC and having to set up an IPSEC VPN, and I'm fully okay with that, however I cannot find a way (since I'm new to leiningen and storm-deploy) to get these new generation machines up and running.

When I try to deploy, I obviously get these errors:

DEBUG compute - >> searching params({imageId=us-east-1/ami-xxxxxxxx, hardwareId=c3.large}) DEBUG compute - >> searching params({imageId=us-east-1/ami-xxxxxxxx, hardwareId=c3.large}) WARN compute - hardware ids that didn't match: [cc1.4xlarge, cg1.4xlarge, cc2.8xlarge, t1.micro, c1.medium, c1.xlarge, m1.large, m1.small, m1.medium, m1.xlarge, m2.xlarge, m2.2xlarge, m2.4xlarge, hi1.4xlarge] java.util.NoSuchElementException: hardwareId(c3.large) not found

gworley3 commented 10 years ago

This is almost certainly because the version of jclouds storm-deploy is using does not know about those. Unfortunately, I think at this point storm-deploy can't come up to the latest version of jclouds without first upgrading to pallet 0.8.0.

vanjos commented 10 years ago

Sorry to be newb about this - but what would the upgrade of pallet to 0.8.0 entail?

gworley3 commented 10 years ago

A lot. Many things changed between 0.7.x and 0.8.x, so it's by no means straightforward.

tbatchelli commented 10 years ago

The pallet-jclouds library that wraps jclouds is compatible with both 0.7 and 0.8 of pallet, so an upgrade to pallet 0.8 is not required, although largely overdue.

Having said this, pallet-jclouds might need an update if the APIs have changed from jclouds 1.5.5.  One quick way to test is to directly upgrade the jclouds dependencies 1.6 or whichever is the jclouds version that has support for the new instances (see below)

We’ll be looking into putting out a newer version of pallet-jclouds soon.

Toni.

The libraries to add into project.clj in storm-deploy in order to force an update to jclouds are:

[org.jclouds/jclouds-all ~jclouds-ver] [org.jclouds.driver/jclouds-sshj ~jclouds-ver] [org.jclouds.driver/jclouds-slf4j ~jclouds-ver] [org.jclouds/jclouds-compute ~jclouds-ver] [org.jclouds/jclouds-blobstore ~jclouds-ver]

where ~jclouds-ver is the version of jclouds that you want.

Also, you might want to update the pallet-jclouds dependency to 1.5.3 (currently 1.5.1)

tbatchelli Sent with Airmail

On February 13, 2014 at 9:17:33 AM, G Gordon Worley III (notifications@github.com) wrote:

A lot. Many things changed between 0.7.x and 0.8.x, so it's by
no means straightforward.

Reply to this email directly or view it on GitHub: https://github.com/nathanmarz/storm-deploy/issues/57#issuecomment-35001850

vanjos commented 10 years ago

Got me really excited --- however, still issues:

--- project.clj --- (defproject storm-deploy "0.0.6-SNAPSHOT" :source-paths ["src/clj"] :test-paths ["test/clj"] :profiles {:dev {:resource-paths ["conf"]}} :aliases {"deploy-storm" ["run" "-m" "backtype.storm.provision"]}

:repositories { "sonatype" "https://oss.sonatype.org/content/repositories/releases" "jclouds-snapshot" "https://oss.sonatype.org/content/repositories/snapshots" }

:dependencies [ [storm "0.5.4"] [commons-codec "1.4"] [org.cloudhoist/pallet "0.7.3"] [org.cloudhoist/pallet-jclouds "1.5.3-SNAPSHOT"] [org.cloudhoist/java "0.5.0"] [org.cloudhoist/git "0.5.0"] [org.cloudhoist/ssh-key "0.5.0"] [org.cloudhoist/automated-admin-user "0.5.0"] [org.cloudhoist/iptables "0.5.0"] [org.cloudhoist/maven "0.5.0"] [org.cloudhoist/zookeeper "0.5.1"] [org.cloudhoist/nagios-config "0.5.0"] [org.cloudhoist/crontab "0.5.0"]

             [com.jcraft/jsch.agentproxy.usocket-jna "0.0.5"]
             [com.jcraft/jsch.agentproxy.usocket-nc "0.0.5"]
             [com.jcraft/jsch.agentproxy.sshagent "0.0.5"]
             [com.jcraft/jsch.agentproxy.pageant "0.0.5"]
             [com.jcraft/jsch.agentproxy.core "0.0.5"]
             [com.jcraft/jsch.agentproxy.jsch "0.0.5"]
             [com.jcraft/jsch "0.1.49"]
             [org.jclouds.driver/jclouds-sshj "1.5.2"]
     [org.jclouds/jclouds-all "1.6.0"]
     [org.jclouds.driver/jclouds-sshj "1.6.0"]
     [org.jclouds.driver/jclouds-slf4j "1.6.0"]
     [org.jclouds/jclouds-compute "1.6.0"]
     [org.jclouds/jclouds-blobstore "1.6.0"]
             [org.jclouds.provider/aws-ec2 "1.5.2"]
             [org.jclouds.provider/aws-s3 "1.5.2"]

             [log4j/log4j "1.2.14"]
             [jvyaml "1.0.0"]]

:dev-dependencies [[swank-clojure "1.2.1"] [org.cloudhoist/pallet-lein "0.5.2"]] :min-lein-version "2.0.0")

Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/reflectinit.class or clojure/reflect.clj on classpath: (jclouds.clj:1) at clojure.lang.Compiler.eval(Compiler.java:5440) at clojure.lang.Compiler.eval(Compiler.java:5415) at clojure.lang.Compiler.load(Compiler.java:5857) at clojure.lang.RT.loadResourceScript(RT.java:340) at clojure.lang.RT.loadResourceScript(RT.java:331) at clojure.lang.RT.load(RT.java:409) at clojure.lang.RT.load(RT.java:381) at clojure.core$load$fn4511.invoke(core.clj:4905) at clojure.core$load.doInvoke(core.clj:4904) at clojure.lang.RestFn.invoke(RestFn.java:409) at clojure.core$load_one.invoke(core.clj:4729) at clojure.core$load_lib.doInvoke(core.clj:4766) at clojure.lang.RestFn.applyTo(RestFn.java:143) at clojure.core$apply.invoke(core.clj:542) at clojure.core$load_libs.doInvoke(core.clj:4800) at clojure.lang.RestFn.applyTo(RestFn.java:138) at clojure.core$apply.invoke(core.clj:542) at clojure.core$require.doInvoke(core.clj:4869) at clojure.lang.RestFn.invoke(RestFn.java:409) at backtype.storm.provision$eval20$loading4410auto__21.invoke(provision.clj:1) at backtype.storm.provision$eval20.invoke(provision.clj:1) at clojure.lang.Compiler.eval(Compiler.java:5424) at clojure.lang.Compiler.eval(Compiler.java:5415) at clojure.lang.Compiler.load(Compiler.java:5857) at clojure.lang.RT.loadResourceScript(RT.java:340) at clojure.lang.RT.loadResourceScript(RT.java:331) at clojure.lang.RT.load(RT.java:409) at clojure.lang.RT.load(RT.java:381) at clojure.core$load$fn4511.invoke(core.clj:4905) at clojure.core$load.doInvoke(core.clj:4904) at clojure.lang.RestFn.invoke(RestFn.java:409) at clojure.core$load_one.invoke(core.clj:4729) at clojure.core$load_lib.doInvoke(core.clj:4766) at clojure.lang.RestFn.applyTo(RestFn.java:143) at clojure.core$apply.invoke(core.clj:542) at clojure.core$load_libs.doInvoke(core.clj:4800) at clojure.lang.RestFn.applyTo(RestFn.java:138) at clojure.core$apply.invoke(core.clj:542) at clojure.core$require.doInvoke(core.clj:4869) at clojure.lang.RestFn.invoke(RestFn.java:409) at user$eval5$fn7.invoke(form-init257033639388524929.clj:1) at user$eval5.invoke(form-init257033639388524929.clj:1) at clojure.lang.Compiler.eval(Compiler.java:5424) at clojure.lang.Compiler.eval(Compiler.java:5415) at clojure.lang.Compiler.load(Compiler.java:5857) at clojure.lang.Compiler.loadFile(Compiler.java:5820) at clojure.main$load_script.invoke(main.clj:221) at clojure.main$init_opt.invoke(main.clj:226) at clojure.main$initialize.invoke(main.clj:254) at clojure.main$null_opt.invoke(main.clj:279) at clojure.main$main.doInvoke(main.clj:354) at clojure.lang.RestFn.invoke(RestFn.java:422) at clojure.lang.Var.invoke(Var.java:369) at clojure.lang.AFn.applyToHelper(AFn.java:165) at clojure.lang.Var.applyTo(Var.java:482) at clojure.main.main(main.java:37) Caused by: java.io.FileNotFoundException: Could not locate clojure/reflectinit.class or clojure/reflect.clj on classpath: at clojure.lang.RT.load(RT.java:412) at clojure.lang.RT.load(RT.java:381) at clojure.core$load$fn4511.invoke(core.clj:4905) at clojure.core$load.doInvoke(core.clj:4904) at clojure.lang.RestFn.invoke(RestFn.java:409) at clojure.core$load_one.invoke(core.clj:4729) at clojure.core$load_lib.doInvoke(core.clj:4766) at clojure.lang.RestFn.applyTo(RestFn.java:143) at clojure.core$apply.invoke(core.clj:542) at clojure.core$load_libs.doInvoke(core.clj:4800) at clojure.lang.RestFn.applyTo(RestFn.java:138) at clojure.core$apply.invoke(core.clj:544) at clojure.core$use.doInvoke(core.clj:4880) at clojure.lang.RestFn.invoke(RestFn.java:458) at pallet.compute.jclouds$eval5815$loading4410auto__5816.invoke(jclouds.clj:1) at pallet.compute.jclouds$eval5815.invoke(jclouds.clj:1) at clojure.lang.Compiler.eval(Compiler.java:5424) ... 55 more

gworley3 commented 10 years ago

I was only able to go as high as pallet-jclouds 1.5.1 and jclouds 1.5.2 when I was actively working on my storm-deploy fork. Haven't revisited since then.

vanjos commented 10 years ago

So even with pallet-jclouds 1.5.1 and jclouds 1.5.2 it doesn't work for me...

tbatchelli commented 10 years ago

I have pushed a branch 'update-clojure-jclouds` that uses the latest jclouds version (1.7.1), and this should allow using all the newer hardware in AWS. Give it a try and report back if you find any issues so I can proceed to merge it into master.