Multinode support for osp

[rmohr]

@cynepco3hahue if you like you can pick that up to add multinode support for our osp cluster. The node needs to be started with --reverse so that the nodes are accessible before the master is up. Then the master discovers all nodes via dns and adds them to its inventory. Finally the master provisiones the nodes. It takes about two minutes to provision a node since all dependencies are already there.

[cynepco3hahue]

@rmohr Hm do we need to clean all master stuff before it? Because we reprovision image as master, I saw you stop and disable master services, does it enough?

[rmohr]

Yes, tried it by hand a few times. No cleaning necessary.

[cynepco3hahue]

I think we must just delete masteronly directory, because the case with NUM_NODE=0, will cover it, all other stuff looks good.

[rmohr]

@cynepco3hahue verified with --nodes 3.

Relevant dns-discover output:

--- node02 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.210/0.210/0.210/0.000 ms
Found node02. Adding it to the inventory.
PING node03 (192.168.66.103) 56(84) bytes of data.
64 bytes from node03 (192.168.66.103): icmp_seq=1 ttl=64 time=0.449 ms

--- node03 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.449/0.449/0.449/0.000 ms
Found node03. Adding it to the inventory.
ping: node04: Name or service not known

All nodes are ready:

cli ssh node01
2018/03/14 09:25:32 Waiting for host: 192.168.66.101:22
2018/03/14 09:25:32 Connected to tcp://192.168.66.101:22
[vagrant@node01 ~]$ sudo /usr/local/bin/oc get nodes
NAME      STATUS    ROLES     AGE       VERSION
node01    Ready     master    11m       v1.9.1+a0ce1bc657
node02    Ready     <none>    37s       v1.9.1+a0ce1bc657
node03    Ready     <none>    37s       v1.9.1+a0ce1bc657

[cynepco3hahue]

@rmohr Cool, then I will merge it and update relevant images.