change zookeeper to use dynamic reconfiguration

[thijsschnitger]

since 3.5.0 zookeeper supports dynamic reconfiguration. this will allow for scaling the number of masters up and down. http://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html

[jrslv]

Can be tested in containers branch.

[smartbit]

ZooKeeper => 3.5 after many years is still in Beta. Zookeeper => 3.5 is not supported with marathon & mesosphere, by the time combination is supported fixing this to Dynamic Reconfiguration would be a great idea.

Zookeeper is not actively developed. "MANTL uses Consul to bootstrap Zookeeper, .... because Zookeeper is quite a headache if you need to add or remove nodes", says Brain Hicks - core developer of MANTL.

Initial steps to make dynamic configuration in scripts/master_config.sh

# populate zoo.cfg
# see http://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html#sc_reconfig_file
echo "writing /etc/zookeeper/conf/zoo.cfg"
sudo sed -i "s/# the port at which the clients will connect/# default standaloneEnabled is true (for legacy reasons), in dynamic configurations it should always be set to false/g" /etc/zookeeper/conf/zoo.cfg
sudo sed -i "s/clientPort=2181/standaloneEnabled=false/g" /etc/zookeeper/conf/zoo.cfg
sudo sh -c "echo dynamicConfigFile=/etc/zookeeper/conf/zoo.cfg.dynamic >> /etc/zookeeper/conf/zoo.cfg"

echo "writing /etc/zookeeper/conf/zoo.cfg.dynamic"
for ((i=1;i<=MASTERCOUNT;i++))
do
  echo "adding server ${i}"
  sudo sh -c "echo server.${i}=${CLUSTERNAME}-mesos-master-$((${i}-1)):2888:3888\;2181 >> /etc/zookeeper/conf/zoo.cfg.dynamic"
done

this is by no means complete nor tested with zookeeper 3.5.0 or higher