Support for bundler/setup

[arielvalentin]

This is just a placeholder for exploration. Right now redstorm sets paths and environment variables in order to resolve where gems are installed. It would be nice to just be able to use bundler/setup along with standalone bundled directories; however I have no clue how much effort it would be to get that working.

[colinsurprenant]

Sure, I'm all open to enhancing the the whole gem/bundling experience, it clearly could work better. relates to #70 last comment.

Also as you mentioned by email, we could also take a look at warbler. There's also maybe JBundler - @mkristian is doing really nice work on the JRuby gem/jar front, maybe he can chime in and/or offer some directions.

[mkristian]

thanx for including me - I just briefly browsed over the RedStorm docu.

for jruby-1.7.14 (next release to come) it will be possible to reduce the installation to

gem install redstorm

and it will install all jar dependencies declared in redstorm.gemspec (assuming all jar dependencies are on maven central). dito it will work via bundler.

this means

redstorm install

will only compile your java files. any particular reason why you did not include the compiled java into the gem ? if there is no particular reason for this other then missing dependencies, then you could include the compiled jruby-bindings already into your gem and the whole redstorm install can go away - judged with my limited understanding.

this jar-dependencies feature in jruby-1.7.14 will called "experimental" but for the above it is up to the task. and it also means that any feedback is welcome and needed ! so if you want to go this road I am happy to prepare an initial PR and it will help this "experimental" feature to mature.

jar-dependencies works without being included in jruby but then you need first install it:

gem install jar-dependencies

and then

gem install redstorm

or

bundle-with-jars install

regarding JBundler and Warbler. warbler is meant for packing warfiles (and jarfiles) and comes with jbundler support. jbundler uses jar-dependencies and both of them depend on ruby-maven for the actual work. ruby-maven is just thin ruby wrapper around maven-3.1.1 and adds a ruby-dsl for maven.

[arielvalentin]

@mkristian the maven dependencies sounds really interesting! I am excited to give that a try. Does that also extend to bundler Gemfiles or will I always have to release a gem?

[mkristian]

if the jar-dependencies is part of jruby (it is on current jruby-1_7 branch) then this works with gem declared inside Gemfiles since bundler obeys the post_install hooks of Rubygems.

[arielvalentin]

But to clarify this feature only works for gems. E.g. I will not be able to add jar deps in the Gemfile of a Rails app. I will have to have a gem with jar deps. Correct?

[mkristian]

not it is NOT an extension of bundler, just for rubygems, jbundler has the intention to "add" jar dependencies along site the Gemfile and here you use a Jarfile for the jar-deps.

or as you mention you can have "meta" gem with only jar dependencies to keep jbundler out of the picture.

[arielvalentin]

Thank you for the clarification!

Ariel Valentin e-mail: ariel@arielvalentin.com website: http://blog.arielvalentin.com skype: ariel.s.valentin twitter: arielvalentin

linkedin: http://www.linkedin.com/profile/view?id=8996534

simplicity communication feedback courage *respect

On Fri, Aug 1, 2014 at 9:56 AM, Christian Meier notifications@github.com wrote:

not it is NOT an extension of bundler, just for rubygems, jbundler has the intention to "add" jar dependencies along site the Gemfile and here you use a Jarfile for the jar-deps.

or as you mention you can have "meta" gem with only jar dependencies to keep jbundler out of the picture.

— Reply to this email directly or view it on GitHub https://github.com/colinsurprenant/redstorm/issues/112#issuecomment-50886707 .

[arielvalentin]

@colinsurprenant Per @mkristian suggestion, is publishing redstorm.jar something that you would be amenable to?

[colinsurprenant]

ah! let me pull in @rtyler who is currently working on refactoring the build & dependency stuff using Graddle.

we were just discussing this on IRC yesterday about publishing a redstorm.jar which would make sense imo.

[rtyler]

@arielvalentin My work in the redstorm project itself can be found in my gradle-experiment branch, with the topology that uses redstorm.jar under redstorm-test-topology.

My primary motivation is to clean up the way in which topologies are generated and reduce the turn-around time it takes on a build of a topology right now. Part of this includes removing some quirky dependency management that's done right now through bundler+ivy.

This work is still highly experimental, so be advised :)

[rtyler]

@arielvalentin If you're interested in checking out some of the approach, I've validated part of it already with my jruby-gradle-plugin. After I bake that a little more I plan on creating a redstorm-gradle-plugin

[mkristian]

when I first look at redstorm there were two things popping up: the install process with compiling java code which I already mentioned.

the second thing which I just want to mention here now is the ivy stuff which I saw and (with my limited view of the world) it look like that jbundler and Jarfile could do the same thing. since I am not at all familiar with redstorm this is just simple thought and going gradle might be the way to go . . .

[arielvalentin]

@mkristian there are a few issues we are trying to address.

• Storm requires a single self contained jar file for deployment. Redstorm uses ant and ivy to assemble the jar.
• Redstorm uses JRuby proxies, which are generated with jrubyc then compiled with javac and packaged in the jar.
• Redstorm users who write custom code may also use additional jars but need to be included in the deployment jar. Redstorm uses ivy to manage those dependencies. Gems like nokogiri, which embed their dependencies are a headache in this case.
• Redstorm packages gems in the jar and sets environment variables to make sure the JRuby Runtime can find them. It uses bundler to figure out what the dependencies are then makes then available to the jar package command

Something else that sprung to mind was maven scopes in Gemspecs. How do we specify that jars are system provided? In our case storm provides things like loggers and their base classes.

[mkristian]

@arielvalentin thanx for the little intro ;) now I see where warbler comes into the picture.

Gems like nokogiri, which embed their dependencies are a headache in this case. once jar-dependencies is part of jruby I would like to contact nokogiri and the likes to at least handle the jars in way we already do with jruby-openssl-0.9.5.gem: use jar-dependencies when present.

Gemspecs and jars you could add :scope => provided - maybe leaving provided jars out unless you need them for compiling somehow.

@rtyler just for your info there is the http://rubygems-proxy.torquebox.org/releases/ repo which has all gems (when the platform is jruby compatible) as maven artifacts and gems with jar-dependencies will have regular jar dependencies, i.e. these gem-artifacts just merge into the dependency tree with both jar and gem dependencies. maybe that is of interest.

[arielvalentin]

@mkristian Where may I read more about the gemspec jar integration? When I use bundler in standalone mode, where are the jars installed?

[arielvalentin]

@rtyler I have not had a chance to look that branch. Is the plan going forward to use gradle instead of rake to build the cluster-topology.jar?

[rtyler]

@arielvalentin The plan right now is for me to experiment with Gradle for managing the builds of redstorm topologies. @colinsurprenant is open to improvements but given I don't have a working prototype it's too early to say what will and will not make it into the redstorm project :)

[mkristian]

@arielvalentin I added some wiki pages: https://github.com/mkristian/jar-dependencies/wiki maybe you find some answers there

[arielvalentin]

That is very helpful. It reads to me like using jar-dependencies to vendor jars is very close to what I wanted when I first asked about it on the jruby mailing list ( http://ruby.11.x6.nabble.com/Maven-dependency-management-td4996934.html). I wonder what the best way to expand vendor/jars for building cluster-topology jar.

Ariel Valentin e-mail: ariel@arielvalentin.com website: http://blog.arielvalentin.com skype: ariel.s.valentin twitter: arielvalentin

linkedin: http://www.linkedin.com/profile/view?id=8996534

simplicity communication feedback courage *respect

On Mon, Aug 18, 2014 at 1:34 AM, Christian Meier notifications@github.com wrote:

@arielvalentin https://github.com/arielvalentin I added some wiki pages: https://github.com/mkristian/jar-dependencies/wiki maybe you find some answers there

— Reply to this email directly or view it on GitHub https://github.com/colinsurprenant/redstorm/issues/112#issuecomment-52453789 .

[mkristian]

@arielvalentin - ah I remember - that basically triggered the jar-dependencies project :)

with warbler+jbundler I am going to go this route: take all the jars from jbundler's Jarfile.lock and copy them into the root of the jar-files or WEB-INF/classes for war-files and use a ruby file "jbundler.rb" to load all those jars into the JRubyClassloader via require_jar the "jbundler.rb" is basically the file I mentioned with gem setup which contains a require_jar for each resolved transitive dependency of the declared jars.

with the jruby-maven-plugins will follow this approach sometime later . . .

[arielvalentin]

I think that will work for war files but not sure about jars. I think what we want is the behavior similar to jar-with-dependencies or shaded jar. Is that something jbundler can do?

[mkristian]

there is some feature to pack an all included jar in jbundler - but it should go away in favour of warbler.

warbler can pack such all included jar - maybe they call it executable jar not sure. any way warbler also packs jar files !!!

keeping the classloader semantic can be important in some case regarding the so called default gems which are part of jruby and which have jar dependencies. currently it is mainly these bouncy-castle jars and with jruby-9000 also the jline jar.

with jruby-9000 you can ONLY use other then the embedded bouncy-castle jars when you do use them via require_jar - the classloader will give preference to those embedded jars over the parent-classloader.

so in the long run any tool embedded jars for the ruby part need to do it via require_jar or require or use the JRubyClassLoader.addURL directly. wanted to ask @rtyler how he is going to do this with the grade-jruby-plugin.

maybe that is not of real issue from the practical POV BUT with bouncy-castle the main security provider of jruby it made become an issue in future.