bundler integration - Githubissues

colinsurprenant commented 12 years ago

Bundler integration to support gems for both local and cluster mode. Related issues #7 and #15

stonegao commented 12 years ago

Any progress? I tried override ENV['GEM_PATH'] and ENV['GEM_HOME'] in boot.rb, but when running on the cluster, bundler still can't figure out the correct path:

2012-02-16 15:20:04 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.4.gemspec]: invalid date format in specification: "2011-08-03 00:00:00.000000000Z"
2012-02-16 15:20:04 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.5.gemspec]: invalid date format in specification: "2011-08-05 00:00:00.000000000Z"
2012-02-16 15:20:04 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.7.gemspec]: invalid date format in specification: "2011-08-08 00:00:00.000000000Z"
2012-02-16 15:20:04 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.9.gemspec]: invalid date format in specification: "2011-08-21 00:00:00.000000000Z"
2012-02-16 15:20:04 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.0.gemspec]: invalid date format in specification: "2011-08-28 00:00:00.000000000Z"
2012-02-16 15:20:05 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.4.gemspec]: invalid date format in specification: "2011-10-31 00:00:00.000000000Z"
2012-02-16 15:20:06 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/rack-protection-1.1.2.gemspec]: invalid date format in specification: "2011-10-01 00:00:00.000000000Z"
2012-02-16 15:20:24 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.4.gemspec]: invalid date format in specification: "2011-08-03 00:00:00.000000000Z"
2012-02-16 15:20:24 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.5.gemspec]: invalid date format in specification: "2011-08-05 00:00:00.000000000Z"
2012-02-16 15:20:24 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.7.gemspec]: invalid date format in specification: "2011-08-08 00:00:00.000000000Z"
2012-02-16 15:20:24 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.9.gemspec]: invalid date format in specification: "2011-08-21 00:00:00.000000000Z"
2012-02-16 15:20:24 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.0.gemspec]: invalid date format in specification: "2011-08-28 00:00:00.000000000Z"
2012-02-16 15:20:24 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.4.gemspec]: invalid date format in specification: "2011-10-31 00:00:00.000000000Z"
2012-02-16 15:20:25 STDIO [ERROR] Invalid gemspec in [file:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/rack-protection-1.1.2.gemspec]: invalid date format in specification: "2011-10-01 00:00:00.000000000Z"
2012-02-16 15:20:31 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.4.gemspec]: invalid date format in specification: "2011-08-03 00:00:00.000000000Z"
2012-02-16 15:20:31 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.5.gemspec]: invalid date format in specification: "2011-08-05 00:00:00.000000000Z"
2012-02-16 15:20:31 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.7.gemspec]: invalid date format in specification: "2011-08-08 00:00:00.000000000Z"
2012-02-16 15:20:31 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.9.gemspec]: invalid date format in specification: "2011-08-21 00:00:00.000000000Z"
2012-02-16 15:20:31 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.0.gemspec]: invalid date format in specification: "2011-08-28 00:00:00.000000000Z"
2012-02-16 15:20:31 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.4.gemspec]: invalid date format in specification: "2011-10-31 00:00:00.000000000Z"
2012-02-16 15:20:33 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/rack-protection-1.1.2.gemspec]: invalid date format in specification: "2011-10-01 00:00:00.000000000Z"
2012-02-16 15:20:40 STDIO [ERROR] Could not find json-1.6.3-java in any of the sources
2012-02-16 15:20:40 STDIO [ERROR] Try running `bundle install`.
2012-02-16 15:20:40 task [ERROR] 
java.lang.RuntimeException: org.jruby.exceptions.MainExitException: aborted
    at redstorm.storm.jruby.JRubyBolt.newProxyBolt(JRubyBolt.java:67)
    at redstorm.storm.jruby.JRubyBolt.prepare(JRubyBolt.java:38)
    at backtype.storm.daemon.task$fn__3298.invoke(task.clj:406)
    at clojure.lang.MultiFn.invoke(MultiFn.java:200)
    at backtype.storm.daemon.task$mk_task$fn__3221.invoke(task.clj:219)
    at backtype.storm.daemon.task$mk_task.invoke(task.clj:218)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361$iter__3409__3413$fn__3414$fn__3415.invoke(worker.clj:182)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361$iter__3409__3413$fn__3414.invoke(worker.clj:182)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:56)
    at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
    at clojure.lang.ChunkedCons.next(ChunkedCons.java:43)
    at clojure.lang.LazySeq.next(LazySeq.java:88)
    at clojure.lang.RT.next(RT.java:560)
    at clojure.core$next.invoke(core.clj:61)
    at clojure.core$dorun.invoke(core.clj:2451)
    at clojure.core$doall.invoke(core.clj:2465)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361.invoke(worker.clj:182)
    at clojure.lang.AFn.applyToHelper(AFn.java:187)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:540)
    at backtype.storm.daemon.worker$fn__3360$mk_worker__3505.doInvoke(worker.clj:78)
    at clojure.lang.RestFn.invoke(RestFn.java:513)
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:268)
    at clojure.lang.AFn.applyToHelper(AFn.java:174)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at backtype.storm.daemon.worker.main(Unknown Source)
Caused by: org.jruby.exceptions.MainExitException: aborted
2012-02-16 15:20:40 util [INFO] Halting process: ("Task died")

stonegao commented 12 years ago

Also tried remove bundler completely and use the approach recommended in redstorm README. I changed the boot.rb to the following :

require 'rubygems'

gem 'mongoid', '= 2.2.4'
gem 'bson', '= 1.3.1'
gem 'bson_ext', '= 1.3.1'
gem 'redis'

require 'red_storm' #, '= 0.4.0'
require 'mongoid'
require 'redis'

It works in local mode. But failed again in cluster mode.

2012-02-17 16:36:19 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.4.gemspec]: invalid date format in specification: "2011-08-03 00:00:00.000000000Z"
2012-02-17 16:36:19 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.5.gemspec]: invalid date format in specification: "2011-08-05 00:00:00.000000000Z"
2012-02-17 16:36:19 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.7.gemspec]: invalid date format in specification: "2011-08-08 00:00:00.000000000Z"
2012-02-17 16:36:20 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.9.gemspec]: invalid date format in specification: "2011-08-21 00:00:00.000000000Z"
2012-02-17 16:36:20 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.0.gemspec]: invalid date format in specification: "2011-08-28 00:00:00.000000000Z"
2012-02-17 16:36:20 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.4.gemspec]: invalid date format in specification: "2011-10-31 00:00:00.000000000Z"
2012-02-17 16:36:21 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/rack-protection-1.1.2.gemspec]: invalid date format in specification: "2011-10-01 00:00:00.000000000Z"
2012-02-17 16:36:52 task [ERROR] 
java.lang.RuntimeException: org.jruby.exceptions.RaiseException: (LoadError) Unable to activate mongo-1.4.0, because bson-1.3.1-java conflicts with bson (= 1.4.0)
    at redstorm.storm.jruby.JRubyBolt.newProxyBolt(JRubyBolt.java:67)
    at redstorm.storm.jruby.JRubyBolt.prepare(JRubyBolt.java:38)
    at backtype.storm.daemon.task$fn__3298.invoke(task.clj:406)
    at clojure.lang.MultiFn.invoke(MultiFn.java:200)
    at backtype.storm.daemon.task$mk_task$fn__3221.invoke(task.clj:219)
    at backtype.storm.daemon.task$mk_task.invoke(task.clj:218)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361$iter__3409__3413$fn__3414$fn__3415.invoke(worker.clj:182)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361$iter__3409__3413$fn__3414.invoke(worker.clj:182)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:56)
    at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
    at clojure.lang.ChunkedCons.next(ChunkedCons.java:43)
    at clojure.lang.LazySeq.next(LazySeq.java:88)
    at clojure.lang.RT.next(RT.java:560)
    at clojure.core$next.invoke(core.clj:61)
    at clojure.core$dorun.invoke(core.clj:2451)
    at clojure.core$doall.invoke(core.clj:2465)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361.invoke(worker.clj:182)
    at clojure.lang.AFn.applyToHelper(AFn.java:187)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:540)
    at backtype.storm.daemon.worker$fn__3360$mk_worker__3505.doInvoke(worker.clj:78)
    at clojure.lang.RestFn.invoke(RestFn.java:513)
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:268)
    at clojure.lang.AFn.applyToHelper(AFn.java:174)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at backtype.storm.daemon.worker.main(Unknown Source)
Caused by: org.jruby.exceptions.RaiseException: (LoadError) Unable to activate mongo-1.4.0, because bson-1.3.1-java conflicts with bson (= 1.4.0)

I set up the cluster env on my dev machine. It seems that redstorm got the wrong GEM_HOME, according to the stack trace it was finding gems in ree not gems in the jar file.

stonegao commented 12 years ago

I changed the source code of JRubyBolt.java and submit the new jar again, there's something wrong with the environment variables GEM_HOME/GEM_PATH:


  @Override
  public void prepare(final Map stormConf, final TopologyContext context, final OutputCollector collector) {
    // create instance of the jruby class here, after deserialization in the workers.
    System.out.println("************ in jrubybot prep****************");
    System.out.println("PATH: " + System.getenv("PATH"));
    System.out.println("GEM_HOME: " + System.getenv("GEM_HOME"));
    System.out.println("GEM_PATH: " + System.getenv("GEM_PATH"));

    System.out.println("************ in jrubybot prep****************");
    _proxyBolt = newProxyBolt(_baseClassPath, _realBoltClassName);
    _proxyBolt.prepare(stormConf, context, collector);
  }

in storm worker log the output is :

2012-02-17 23:25:36 STDIO [INFO] ************ in jrubybot prep****************
2012-02-17 23:25:36 STDIO [INFO] PATH: /Users/stone/.cljr/bin:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/bin:/Users/stone/.rvm/gems/ree-1.8.7-2010.02@global/bin:/Users/stone/.rvm/rubies/ree-1.8.7-2010.02/bin:/Users/stone/.rvm/bin:/Users/stone/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin:/usr/X11/bin
2012-02-17 23:25:36 STDIO [INFO] GEM_HOME: /Users/stone/.rvm/gems/ree-1.8.7-2010.02
2012-02-17 23:25:36 STDIO [INFO] GEM_PATH: /Users/stone/.rvm/gems/ree-1.8.7-2010.02:/Users/stone/.rvm/gems/ree-1.8.7-2010.02@global
2012-02-17 23:25:36 STDIO [INFO] ************ in jrubybot prep****************
2012-02-17 23:26:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.4.gemspec]: invalid date format in specification: "2011-08-03 00:00:00.000000000Z"
2012-02-17 23:26:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.5.gemspec]: invalid date format in specification: "2011-08-05 00:00:00.000000000Z"
2012-02-17 23:26:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.7.gemspec]: invalid date format in specification: "2011-08-08 00:00:00.000000000Z"
2012-02-17 23:26:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.9.gemspec]: invalid date format in specification: "2011-08-21 00:00:00.000000000Z"
2012-02-17 23:26:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.0.gemspec]: invalid date format in specification: "2011-08-28 00:00:00.000000000Z"
2012-02-17 23:26:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.4.gemspec]: invalid date format in specification: "2011-10-31 00:00:00.000000000Z"
2012-02-17 23:26:10 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/rack-protection-1.1.2.gemspec]: invalid date format in specification: "2011-10-01 00:00:00.000000000Z"
2012-02-17 23:26:41 task [ERROR] 
java.lang.RuntimeException: org.jruby.exceptions.RaiseException: (LoadError) Unable to activate mongo-1.4.0, because bson-1.3.1-java conflicts with bson (= 1.4.0)
    at redstorm.storm.jruby.JRubyBolt.newProxyBolt(JRubyBolt.java:73)
    at redstorm.storm.jruby.JRubyBolt.prepare(JRubyBolt.java:44)
    at backtype.storm.daemon.task$fn__3298.invoke(task.clj:406)

stonegao commented 12 years ago

FYI: warbler can also used to package ruby applications https://github.com/jruby/warbler

colinsurprenant commented 12 years ago

Stone,

I spent some time yesterday on Bundler and I am stuck where Bundler does not find its gems when calling setup. Basically, the idea is to include the bundler gem, the Gemfile and the bundled gems in the jar. I was able to package bundler, have it find its Gemfile but there is still a path issue when bundler calls setup. This is all a bit tricky since the environment & paths are inside the jar file. Still working on this.

Thanks for your investigations, this is helpful. There is clearly an environment issue. On my side, when I did the gem testing, my storm cluster was started in a "clean" environment, without any Ruby and RVM setup. Is this something your could try?

colinsurprenant commented 12 years ago

Good news! I think I just cracked that nut :) I was able to make everything work with bundler. Now I need to cleanup and refactor. I should be able to push a branch soon for you to test in your environment.

stonegao commented 12 years ago

Thanks & Great news. Looking forward to your commits.

stonegao commented 12 years ago

FYI

I tried start storm like unset GEM_PATH && unset GEM_HOME && bin/storm supervisor as this post suggested ( http://blog.karthiks.in/2010/04/enforce-jruby-to-use-gems-inside-jruby.html ) , it no longer find gems for ree now, but still complain that Could not find multi_json :

2012-02-18 18:58:06 STDIO [INFO] ************ in jrubybot prep****************
2012-02-18 18:58:06 STDIO [INFO] PATH: /Users/stone/.cljr/bin:/Users/stone/.rvm/gems/ree-1.8.7-2010.02/bin:/Users/stone/.rvm/gems/ree-1.8.7-2010.02@global/bin:/Users/stone/.rvm/rubies/ree-1.8.7-2010.02/bin:/Users/stone/.rvm/bin:/Users/stone/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin:/usr/X11/bin
2012-02-18 18:58:06 STDIO [INFO] GEM_HOME: null
2012-02-18 18:58:06 STDIO [INFO] GEM_PATH: null
2012-02-18 18:58:06 STDIO [INFO] ************ in jrubybot prep****************
2012-02-18 18:58:11 task [ERROR] 
java.lang.RuntimeException: org.jruby.exceptions.RaiseException: (LoadError) Could not find multi_json (~> 1.0) amongst [activemodel-3.2.1, activesupport-3.2.1, activesupport-3.0.3, bson-1.3.1-java, bson_ext-1.3.1, builder-3.0.0, i18n-0.6.0, json-1.6.5-java, mongo-1.3.1, mongoid-2.2.4, mongoid-collection-sharding-0.0.3, rack-1.4.1, rack-protection-1.2.0, rake-0.9.2.2, rake-0.8.7, redis-2.2.2, redis-namespace-1.0.3, redstorm-0.4.0, resque-1.19.0, sinatra-1.3.2, sources-0.0.1, vegas-0.1.11]
    at redstorm.storm.jruby.JRubyBolt.newProxyBolt(JRubyBolt.java:73)
    at redstorm.storm.jruby.JRubyBolt.prepare(JRubyBolt.java:44)
    at backtype.storm.daemon.task$fn__3298.invoke(task.clj:406)
    at clojure.lang.MultiFn.invoke(MultiFn.java:200)
    at backtype.storm.daemon.task$mk_task$fn__3221.invoke(task.clj:219)
    at backtype.storm.daemon.task$mk_task.invoke(task.clj:218)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361$iter__3409__3413$fn__3414$fn__3415.invoke(worker.clj:182)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361$iter__3409__3413$fn__3414.invoke(worker.clj:182)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:56)
    at clojure.lang.RT.seq(RT.java:450)
    at clojure.core$seq.invoke(core.clj:122)
    at clojure.core$dorun.invoke(core.clj:2450)
    at clojure.core$doall.invoke(core.clj:2465)
    at backtype.storm.daemon.worker$fn__3360$exec_fn__994__auto____3361.invoke(worker.clj:182)
    at clojure.lang.AFn.applyToHelper(AFn.java:187)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:540)
    at backtype.storm.daemon.worker$fn__3360$mk_worker__3505.doInvoke(worker.clj:78)
    at clojure.lang.RestFn.invoke(RestFn.java:513)
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:268)
    at clojure.lang.AFn.applyToHelper(AFn.java:174)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at backtype.storm.daemon.worker.main(Unknown Source)
Caused by: org.jruby.exceptions.RaiseException: (LoadError) Could not find multi_json (~> 1.0) amongst [activemodel-3.2.1, activesupport-3.2.1, activesupport-3.0.3, bson-1.3.1-java, bson_ext-1.3.1, builder-3.0.0, i18n-0.6.0, json-1.6.5-java, mongo-1.3.1, mongoid-2.2.4, mongoid-collection-sharding-0.0.3, rack-1.4.1, rack-protection-1.2.0, rake-0.9.2.2, rake-0.8.7, redis-2.2.2, redis-namespace-1.0.3, redstorm-0.4.0, resque-1.19.0, sinatra-1.3.2, sources-0.0.1, vegas-0.1.11]

colinsurprenant commented 12 years ago

It seems multi_json is not installed?

(LoadError) Could not find multi_json (~> 1.0) amongst [activemodel-3.2.1, activesupport-3.2.1, activesupport-3.0.3, bson-1.3.1-java, bson_ext-1.3.1, builder-3.0.0, i18n-0.6.0, json-1.6.5-java, mongo-1.3.1, mongoid-2.2.4, mongoid-collection-sharding-0.0.3, rack-1.4.1, rack-protection-1.2.0, rake-0.9.2.2, rake-0.8.7, redis-2.2.2, redis-namespace-1.0.3, redstorm-0.4.0, resque-1.19.0, sinatra-1.3.2, sources-0.0.1, vegas-0.1.11]

stonegao commented 12 years ago

It does installed and get packaged in the jar

colinsurprenant commented 12 years ago

I am about to finally push my changes so I think it's best you try with that instead of trying to figure out this particular problem...

colinsurprenant commented 12 years ago

Ok, I just pushed branch "bundler". Can you try it with your project?

build gem (0.4.1) and install it in your project
i strongly suggest you wipe the target directory. reinstall redstorm in your project
redstorm install
in your project, install bundler and rake gems into target dir using:
gem install bundler -i target/gems/gems
gem install rake -i target/gems/gems
install the bundled gems into target dir using:
bundle install --path target/gems/bundler
package the cluster jar
redstorm jar

Let me know how this works out.

Colin

colinsurprenant commented 12 years ago

Forgot to say that you should have the following require order in the topology:

require 'rubygems'
require 'red_storm'
require 'bundler/setup'

you can look at examples/simple/redis_word_count_topology.rb which works with bundler.

stonegao commented 12 years ago

The code in bundler branch works ok. thanks for your effort!

But it still print the following msg (this does not prevent redstorm topology from working, just the msg).

2012-03-01 11:45:07 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.4.gemspec]: invalid date format in specification: "2011-08-03 00:00:00.000000000Z"
2012-03-01 11:45:08 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.5.gemspec]: invalid date format in specification: "2011-08-05 00:00:00.000000000Z"
2012-03-01 11:45:08 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.7.gemspec]: invalid date format in specification: "2011-08-08 00:00:00.000000000Z"
2012-03-01 11:45:08 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.1.9.gemspec]: invalid date format in specification: "2011-08-21 00:00:00.000000000Z"
2012-03-01 11:45:08 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.0.gemspec]: invalid date format in specification: "2011-08-28 00:00:00.000000000Z"
2012-03-01 11:45:08 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/mongoid-2.2.4.gemspec]: invalid date format in specification: "2011-10-31 00:00:00.000000000Z"
2012-03-01 11:45:09 STDIO [ERROR] Invalid gemspec in [/Users/stone/.rvm/gems/ree-1.8.7-2010.02/specifications/rack-protection-1.1.2.gemspec]: invalid date format in specification: "2011-10-01 00:00:00.000000000Z"

colinsurprenant commented 12 years ago

I have reworked the bundler support and also fixed the 1.8/1.9 Ruby mode. Can you try your topology with the current master?

colinsurprenant commented 12 years ago

I think the problem with the invalid gemspec is related to 1.8/1.9 mixup with gems. It is important to specify the same 1.8 or 1.9 mode throughout the whole workflow to avoid incompatibilities.

colinsurprenant commented 12 years ago

closing this issue. all bundling & gem support working in master and about to be released in gem v0.5.0.

colinsurprenant / redstorm

bundler integration #16