[HALON-873] Multi-node `h0` script

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: vvv

CLI differences from the traditional (single-node) h0 script:

• delete start/stop/install/uninstall commands;
• add init/fini commands;
• get initial cluster configuration from YAML file. Path to this file is taken from M0_CLUSTER environment variable (defaults to ~/.m0-cluster.yaml). See m0genfacts -h for the format description.

Sample configuration

(Tested with m0vg setup.)

$ cat ~/.m0-cluster.yaml
confds: [ cmu.local ]
ssus:
  - host:  ssu1.local
    disks: /dev/sd[b-g]
  - host:  ssu2.local
    disks: /dev/sd[b-g]
clients:     [ client1.local ]
clovis-apps: [ client1.local ]

Known issues

• [x] 1. Commit 1be8da5ad3644ea4838151b84c6a1f7a070d9463 breaks Jenkins CI:

  > cd /root/build//halon && ./scripts/h0 setup
  Invalid command: setup
  Type `h0 help' for usage.

  TODO: Update Xperior script.

• [x] 2. init fails:

  $ ~/h/scripts/h0-new || echo FAIL $?
  [...]
  --->  Getting facts about 'client1.local' node
  m0genfacts: ERROR: `facter` is not found on host '172.16.1.234', try to install it with `yum install facter`
  [...]
  FAIL 1

  QUESTION: @chumakd, is it necessary for facter to be installed on client1?

• [ ] 3. TODO: Rewrite system tests using h0-new.

• [x] 4. This error message should be suppressed:

  
  $ ~/h/scripts/h0-new init
  [...]

• [mpdsh:269] pdsh -S -f1 -w client1.local,ssu1.local,ssu2.local,cmu.local sudo systemctl start halon-cleanup client1: Job for halon-cleanup.service failed because the control process exited with error code. See "systemctl status halon-cleanup.service" and "journalctl -xe" for details. pdsh@cmu: client1: ssh exited with exit code 1

• [erase_cluster_data:335] true [...]

  
  TODO: Find out why `halon-cleanup` fails on `client1`. (Bonus points for rewriting [halon-cleanup](https://github.com/seagate-ssg/halon/blob/master/mero-halon/src/halon-cleanup/Main.hs) in bash.)

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: vvv

I'd prefer not to make h0 dependent on m0vg if we can help it.

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: chumakd

On VirtualBox, the first network interface (eth0) is not accessible outside of VM. The second interface is configured by m0vg specifically to be accessible among all VMs.

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: chumakd

is it necessary for facter to be installed on client1?

@vvv It is. Though, it is installed with m0vg up|provision automatically on every VM. If facter is missing on your client1 VM that means it must've been provisioned incorrectly.

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: chumakd

It was needed mostly because of m0provision:cmd_genfacts() func that does sudo m0genfacts. I think current implementation of m0genfacts does sudo internally so the script itself doesn't need to be called with sudo any more. I suppose we can drop sudo in front of m0genfacts and don't generate /root/.ssh/known_hosts, but that should tested.

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: chumakd

done, I've restarted Jenkins job: http://jenkins.mero.colo.seagate.com:8080/job/halon_github_trigger/138/

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: vvv

This breaks Jenkins CI. h0 setup call should be removed from Xperior script.

[1468ca0b-2a64-4fb4-8e52-ea5806644b4c]

Created by: vvv

@chumakd Is modification of /root/.ssh/known_hosts really needed?