hpc-collab (sometimes: hpc-colab)

This project provides provisioned HPC cluster models using underlying virtualization mechanisms.

The purpose of this project is to provide a common baseline for repeatable HPC experiments. This has been used for education, distributed collaboration, tool development colaboration, failure signature discovery, local HPC debugging and cluster configuration comparisons, enabled by construction and use of short-lived and common baseline hpc cluster models. In short, extend the "systems as cattle not pets" [1] [2] analogy into the realm of "clusters as cattle, not pets."

In effect, this project automates, replaces and enables customized recipes for manually-executed
cluster component construction, installation, configuration and verification processes.

The initial release requires local enablers: gmake, vagrant and virtualbox and, if specified, libvirt, and its accompanying vagrant-libvirt plugin. Graphviz, doxygen and vmtouch are recommended, but not required. A local copy of the clever bash-doxygen sed filter is included. Lighterweight and multi-node mechanisms are welcomed and planned.

Two representative HPC cluster recipes are provided. Cluster recipes are in the clusters directory. Presently, recipes generate clusters local to the installation host, only.

vc is a virtual machine-based cluster, configured with the service-factored following nodes:

• vcfs - provides file storage to other cluster nodes, including common configuration and logs (slurm, rsyslog)
• vcsvc - provides common in-bound services such as DNS, NTP, SYSLOG
• vcbuild - configured with a larger share of RAM and cpu cores, compilation HPC partition,
  builds software (slurm, lustre) as it is brought up, if necessary
• vcdb - provides mysql service, holds the slurm scheduling database daemon
• vcaltdb - provides an alternate mysql db, configured as a replicant of the primary data base
• vcsched - provides the slurm controller and scheduler service
• vc[1-2] - computational nodes
• vclogin - front-end/login node, provides vc-cluster job submission services
• vcgate - externally-accessible node via bridged 3rd interface

vx is a minimal, conjoined virtual-machine cluster, dependent upon vc-provided services and nodes

• vxsched - provides the slurm controller and scheduler service, dependent upon vcdb, vcbuild, vcsvc, vcfs
• vx[1-2] - computational nodes
• vxlogin - front-end/login node, provides vx-cluster job submission services

Security Note

This software constructs models of production clusters which include enabled and enforcing security features. However, since the cluster models are constructed to automate and compare cluster experiments, these cluster recipes are not in themselves secure. Different cluster recipes would be required to construct cluster images with security guarantees.

To start

Set the BASE directory in bin/setpath.{c}sh. The default setting is the output of pwd, often $HOME/hpc-collab or $HOME/hpc-collab-<branch-name>.
~~~ cd hpc-collab [csh] % source bin/setpath.csh [bash/zsh] $ . bin/setpath.sh % make -C clusters/vc Vagrantfile % make -C clusters/vx Vagrantfile % ln -s /scratch/tarballs <--- assuming a separate, larger /scratch partition ~~~

Consider setting the value clusters/common/flag/PREFERRED_REPO to your nearest rsync reposistory. Alternatively, reorder the file requires/ingest/repos. The last line in the file will be the preferred repository, by default.

Then make prereq to sanity check that there is sufficient storage to host this set of cluster recipes and to construct the appropriate Vagrantfiles for the local environment. Point hpc-collab/tarballs, $HOME/VirtualBox VMs and /var/lib/libvirt/images at a separate partition with more storage, if needed. Examine requires/sw/* to determine whether additional software needs to be installed onto the host, such as the vagrant libvirt plugin. The vagrant-proxyconf plugin may be necessary if individual nodes require a proxy server to establish yum installation connections.

The virtualization provider is set in clusters/common/Vagrantfile.d/cfg.vm.providers/default_provider. By default it is virtualbox. The configuration flag clusters/common/flag/NO_NFS is set. When changing these settings, it may be necessary to rm clusters/common/{vc,vx}/.regenerated. The Vagrantfile is dynamically composed based on these configuration parameters. Each virtualization provider uses different ranges of private IP address space for its own cluster-internal private network. For convenience, cat clusters/vc/common/etc/hosts >> /etc/hosts, when regenerating the various configuration files for each virtualization provider.

Virtualbox may be slower than libvirt provisioning, especially if NO_NFS is set, although it is more consistent and reliable and does not require administrative privileges for a local NFS server.

The default configuration settings of virtualbox and NO_NFS require no elevated privileges. Generally, this combination has the fewest installation and compatibility issues.

Cluster recipes are driven by configuration stored in skeleton file systems. Vagrant Vagrantfile and GNU make rules ingest the settings from the cfg/<nodenames> directories.

In the interest of documentation that matches actual code, preliminary work has been done with bash-doxygen.sed.

Make systematizes the dependencies and invocations. In order to avoid all of its arguments, convenience aliases are created in the setpath.csh and setpath.sh shell-specific files. A future implementation will convert these to Modulefiles.

• cd clusters/vc; make Vagrantfile - to construct initial Vagrantfile
  
• make prereq - simplistic check of underlying prerequisites
• make provision - identical to 'make up'
• make show - shows virtual cluster nodes and their state
• make up - provisions virtual cluster nodes
• make unprovision - destroys virtual clusters, their nodes and underlying resources

Aliases are provided by the setpath helper. When using them as recommended, the appropriate Makefile is set so that one need not be in a cluster directory.