Launching multiple supervisors within a single host

[anfeng]

Currently, supervisors are launched with pre-configured port #. In many use cases, we want to launch multiple supervisors with a single storm.yaml. That will only work if these supervisors were launched on separated hosts.

We should enhance storm core to allow ports to be discovered dynamically. That will empower storm-yarn to launch multiple supervisors even in a single host.

[viirya]

Is launching multiple supervisors on a single host making sense? Since supervisor is responsible to start and stop worker processes, the ideal case is to have only one supervisor on a host as the storm does now. That is like why we have only one nodemanger on a host to oversee container management, etc. Or we should ask what the benefit is in allowing multiple supervisors on a single host?

[revans2]

I agree, that is the ideal case, but YARN does not currently give any clean interface to request specific ports be available on that machine like Mesos does. There was some discussion on YARN about adding it in, but there was push back on it because Mesos does a best effort but cannot truly guarantee that a port is free.

YARN also does not give a clean interface to restrict multiple containers being assigned to the same machine. It is possible but it involves the AM throwing away containers that happen to show up on the same machine. Even if we did that it would still be possible for separate supervisors from different storm clusters running on the same YARN cluster to end up running on the same box.

[pinipints]

when we add new supervisor via storm-yarn, is it possible to provide the configuration for the new supervisor. Example, I have a very large node and I want to have 8 workers on that node rather than 4, is it possible?