dsKarthick
commented
8 years ago @JessicaLHartog +1 for topology.mesos.scheduler - Thats the whole reason behind creating Scheduling Framework described in https://github.com/mesos/storm/pull/93.
I would like to propose IsolationScheduler along with your multifarious schedulers - such a scheduler would dedicate set of hosts for a particular topology while also ensuring no other worker gets scheduled on to those dedicated hosts. This would be helpful for debugging nefarious topologies.
erikdw
bigdata-user
Sometimes the approach of "let Storm do its own thing" when it comes to scheduling worker slots can lead to interesting scenarios:
(1) A topology may wish to enable running agents in the JVM which listen on predefined ports (such as JDWP debugging or JMX remote). In such a scenario if the port is statically defined somewhere, then two workers on the same host will stall the topology. For example, if we pass
-Dcom.sun.management.jmxremote.port=9111as one of the java childopts, we will see an exception like the one below when trying to bring up the second worker:(2) Two worker processes on the same machine for the same topology can lead to inconsistent behavior. In a very resource-hungry topology predictability about how workers are scheduled across various hosts may yield better results when it comes to verifying behavior. For example, suppose a topology writes to disk a lot. Having two workers on the same host will increase the I/O on the worker, and can cause an inability to write heartbeat messages, bringing the worker offline. When these workers are rescheduled, if they are scheduled onto two new hosts, then the topology will likely not cause a crashing of the worker.
(3) Tuning the configuration of a topology for various elements like the number of required workers as well as the number of executors per topology component can be made difficult if the first time a topology is submitted two workers are scheduled on the same host, and after some tweaking of configuration options the workers are spread across more hosts (and vice versa).
To make these behaviors (and others) more predictable it would be nice if there were an option like
topology.mesos.schedulerthat can define a TopologyScheduler. The purpose of the TopologyScheduler would be to define worker spreading across hosts. Some options that may be useful are:Default- the current "let Storm do its own thing" behaviorOneWorkerPerHost- worker slots for the same topology may not use offers from the same hostMinimumNumberOfHosts- all worker slots are scheduled on the same host (or on as few hosts as possible), attempting to maximize the number of worker slots on each hostAdditional options would also then be possible should a need for them arise.
This also relates to Issue 158: enhance scheduling to act more predictably