search

yahoo / storm-yarn

Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.
Other
417 stars 161 forks source link

Proposal: Merge storm-yarn into storm source tree #46

Open clockfly opened 11 years ago

clockfly commented 11 years ago

I think it is a good idea to merge storm-yarn into storm source tree. This approach have been adopted by spark. spark-yarn is a internal part of spark.

Advantages:

  1. More easy to manage the dependance.
  2. We can finally throw away the zmq code and dependance for build and test.
  3. In storm-yarn, we don't need to install both, don't need to manage STORM_HOME in storm-yarn anymore. There will be single installation, and this can be build into single rpm package. Very convenient for management.
  4. No longer need local storm.zip package. We can set the jars(storm.jar storm-yarn.jar) and libraries as local resources, and use this to start container on another machine.

Steps:

  1. First need to change the groupId to "storm"
  2. Better we need to use pom to manage storm, so that we can more easily aggregate storm-yarn as a module.
  3. Add interface in storm-yarn scripts, so that we can submit storm job with storm-yarn script file directly (no need to getStormConfig)
  4. We should not try to copy storm.zip into HDFS and use that zip to deploy. Instead, we can just add lib and jars into local resource. Otherwise, there are two copy of storm. One is in STORM_HOME, the other is in storm.zip.
clockfly commented 11 years ago

After merging, the binary layout of storm is like this:

/etc/storm/conf /usr/bin/storm-yarn --> /usr/lib/storm/bin/storm-yarn /usr/lib/storm/ Storm-core.jar Storm-netty.jar Storm-yarn.jar /usr/lib/storm/bin/ Storm-yarn Storm /usr/lib/storm/lib/ dependant-xx.jar .. dependant-yy.jar

I think the layout is much cleaner than current storm-yarn deployment.

revans2 commented 11 years ago

Totally agree. Storm has a proposal up to become an Apache incubator project. The plan was to wait until it is part of the incubator and then work with Nathan about brining in it.

anfeng commented 11 years ago

I don't think that storm-yarn should be merged into storm at this point. Our plan is to have storm-yarn and storm-mesos as a subproject in apache

Andy Feng

Sent from my iPhone

On Sep 18, 2013, at 9:36 AM, Sean Zhong notifications@github.com wrote:

After merging, the binary layout of storm is like this:

/etc/storm/conf /usr/bin/storm-yarn --> /usr/lib/storm/bin/storm-yarn /usr/lib/storm/ Storm-core.jar Storm-netty.jar Storm-yarn.jar /usr/lib/storm/bin/ Storm-yarn Storm /usr/lib/storm/lib/ dependant-xx.jar .. dependant-yy.jar

I think the layout is much cleaner than current storm-yarn deployment.

— Reply to this email directly or view it on GitHub.