Open drewrobb opened 8 years ago
@drewrobb : I'm not yet familiar with the HA Nimbus implementation upcoming in Storm 0.11 1.0. I found reference to the design doc though so will need to read through that to be able to answer you.
@drewrobb it seems 0.11.0 is now storm 1.0.0. Storm 1.0.0 is going to be released the next weeks or so, so we should start checking support for 1.0.0 .
I'm really excited for Storm HA support.
@drewrobb @erikdw had you have time to check on this ? Thanks
Allong those lines we'll need to just ensure 1.0.0 will run as there may be breaking changes (as in 0.10.0). Fun as I haven't fully migrated to 0.10.x yet...
@DarinJ you might want to start migrating to 1.0.0 :D
@salimane : ah, I missed that change from 0.11.0 to 1.0.0 (I was assuming that both still existed, which would have been ultra-confusing with 2.0.0 also in the works!).
I haven't looked at 1.0.0 at all, except reading some random code reviews and emails on the storm user and dev lists.
I'm in a similar boat to @DarinJ, except my boat might be further from shore:
@erikdw I was also bitten during testing by the migration from "logback" to "log4j 2" but, luckily, we are using puppet for server provisioning https://github.com/meltwater/puppet-storm. So there would be only one place to update.
this just became real as storm version 1.0.0 was just released today https://storm.apache.org/2016/04/12/storm100-released.html :)
@salimane : ohh no. :) Thanks for pointing this out.
Via PR #177 we have added support for storm 1.0+. We haven't done anything specifically for supporting HA Nimbus in this project yet, but now we have the support for the version of storm that includes it.
Jumping the gun here, but storm ~0.11~ 1.0 supports HA nimbus when released: https://github.com/apache/storm/pull/354. It would be extremely nice if the mesos storm framework supported this directly. I see two options:
A) Just allow the user to run multiple
storm-mesos nimbus
processes directly, and configure the HA options separately if desired. Make sure that the framework leader and associated state follows the elected nimbus master.B) Split the framework scheduler and the actual nimbus process, so that the scheduler will launch N nimbus instances as mesos tasks.
Seems like choice A) would be a lot simpler to implement, but I'm not really sure. A) would certainly not be any more difficult to deploy assuming that the user is already using a framework like marathon that could provision multiple instances of nimbus (and would not re-implement this responsibility).