This repository contains playbooks and documentation to deploy stacks of virtual machines working together. Most of these stacks are virtual Linux HPC clusters, which can be used as collaborative, analytical sandboxes. All production clusters were named after robots that appear in the animated sitcom Futurama. Test/development clusters were named after other robots.
The main ingredients for (deploying) these clusters:
The master and develop branches of this repo are protected; updates can only be merged into these branches using reviewed pull requests.
Once a while we create releases, which are versioned using the format YY.MM.v
where:
YY
is the year of releaseMM
is the month of releasev
is the first release in that month and year. Hence it is not the day of the month.E.g. 19.01.1
is the first release in January 2019.
We follow the Python PEP8 naming conventions for variable names, function names, etc.
This repo currently contains code and configs for the following clusters:
Deployment and functional administration of all clusters is a joined effort of the Genomics Coordination Center (GCC) and the Center for Information Technology (CIT) from the University Medical Center and University of Groningen, in collaboration with ELIXIR compute platform, EXCELERATE, EU-Solve-RD, European Joint Programme on Rare Diseases and CORBEL projects.
The clusters are composed of the following type of machines:
The clusters use the following types of storage systems / folders:
Filesystem/Folder | Shared/Local | Backups | Mounted on | Purpose/Features |
---|---|---|---|---|
/home/${home}/ | Shared | Yes | UIs, DAIs, SAIs, CNs | Only for personal preferences: small data == tiny quota. |
/groups/${group}/prm[0-9]/ | Shared | Yes | UIs, DAIs | permanent storage folders: for rawdata or final results that need to be stored for the mid/long term. |
/groups/${group}/tmp[0-9]/ | Shared | No | UIs, DAIs, CNs | temporary storage folders: for staged rawdata and intermediate results on compute nodes that only need to be stored for the short term. |
/groups/${group}/scr[0-9]/ | Local | No | Some UIs | scratch storage folders: same as tmp, but local storage as opposed to shared storage. Optional and available on all UIs. |
/local/${slurm_job_id} | Local | No | CNs | Local storage on compute nodes only available during job execution. Hence folders are automatically created when a job starts and deleted when it finishes. |
/mnt/${complete_filesystem} | Shared | Mixed | SAIs | Complete file systems, which may contain various home , prm , tmp or scr dirs. |
Some other stacks of related machines are:
docs_library
: web servers hosting documentation.jenkins_server
: Continues Integration testing server.Deploying a fully functional stack of virtual machines from scratch involves the following steps:
mkdir -p ${HOME}/git/
cd ${HOME}/git/
git clone https://github.com/rug-cit-hpc/league-of-robots.git
cd league-of-robots
#
# For older openstacksdk < 0.99 we need the ansible openstack collection 1.x.
# For newer openstacksdk > 1.00 we need the ansible openstack collection 2.x.
#
openstacksdk_major_version='3' # Change for older OpenStack SDK.
#
# Create Python virtual environment (once)
#
python3 -m venv openstacksdk-${openstacksdk_major_version:-3}.venv
#
# Activate virtual environment.
#
source openstacksdk-${openstacksdk_major_version:-3}.venv/bin/activate
#
# Install OpenStack SDK (once) and other python packages.
#
pip3 install --upgrade pip
pip3 install wheel
pip3 install setuptools # No longer part of default Python >= 3.12.x, but we need it.
if [[ "${openstacksdk_major_version:-3}" -eq 0 ]]; then
pip3 install "openstacksdk<0.99"
else
pip3 install "openstacksdk==${openstacksdk_major_version:-3}.*"
fi
pip3 install openstackclient
pip3 install ruamel.yaml
pip3 install netaddr
#
# Package dnspython is required for Ansible lookup plugin community.general.dig
#
pip3 install dnspython
#
# On macOS only to prevent this error:
# crypt.crypt not supported on Mac OS X/Darwin, install passlib python module.
#
pip3 install passlib
#
# Optional: install Ansible and the Ansible linter with pip.
# You may skip this step if you already installed Ansible by other means.
# E.g. with HomeBrew on macOS, with yum or dnf on Linux, etc.
#
# Ansible core 2.16 from Ansible 9.x is latest version compatible with Mitogen.
#
pip3 install 'ansible<10' # For running playbooks on your local laptop as Ansible control host.
pip3 install 'ansible<6' # For running playbooks directly on chaperone machines running RHEL8.
pip3 install ansible-lint
#
# Optional: install Mitogen with pip.
# Mitogen provides an optional strategy plugin that makes playbooks a lot (up to 7 times!) faster.
# See https://mitogen.networkgenomics.com/ansible_detailed.html
#
pip3 install mitogen
source openstacksdk-${openstacksdk_major_version:-3}.venv/bin/activate
export ANSIBLE_ROLES_PATH="${VIRTUAL_ENV}/ansible/ansible_roles/:"
export ANSIBLE_COLLECTIONS_PATH="${VIRTUAL_ENV}/ansible/:"
ansible-galaxy install -r requirements-${openstacksdk_major_version:-3}.yml
Note: the default location where these dependencies will get installed with the ansible-galaxy install
command is ${HOME}/.ansible/
,
which may conflict with versions of roles and collections required for other repos.
Therefore we set ANSIBLE_ROLES_PATH
and ANSIBLE_COLLECTIONS_PATH
to use a custom path for the dependencies inside the virtual environment we'll use for this repo.
vault_pass.txt
.The vault password is used to encrypt/decrypt the secrets.yml
file per stack_name,
which will be created in the next step if you do not already have one.
In addition a second vault passwd is used for various files in group_vars/all/
and which contain settings that are the same for all stacks.
If you have multiple stacks with their own vault passwd you will have multiple vault password files.
The pattern .vault*
is part of .gitignore
, so if you put the vault passwd files in the .vault/
subdir,
they will not accidentally get committed to the repo.
.vault/vault_pass.txt.[stack_name|all]
, use the following oneliner:
LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w60 | head -n1 > .vault/vault_pass.txt.[stack_name|all]
.vault/vault_pass.txt.[stack_name|all]
and use a text editor to add the password..vault/
subdir and it's content is private:
chmod -R go-rwx .vault/
To create a new stack you will need group_vars
and a static inventory for that stack:
static_inventories/*.yml
files for existing stacks for examples.static_inventories/[stack_name].yml
.group_vars/[stack_name]/
folder with a vars.yml
.vars.yml
file in group_vars/template/
.secrets.yml
with new random passwords for the various daemons/components and encrypt this new secrets.yml
file:
#
# Activate Python virtual env created in step 0.
#
source openstacksdk-${openstacksdk_major_version:-3}.venv/bin/activate
#
# Configure this repo for a specific cluster.
# This will set required ENVIRONMENT variables including
# ANSIBLE_VAULT_IDENTITY_LIST='all@.vault/vault_pass.txt.all, [stack_name]@.vault/vault_pass.txt.[stack_name]'
#
. ./lor-init
lor-config [stack_prefix]
#
#
# Create new secrets.yml file based on a template and encrypt it with the vault password.
#
./generate_secrets.py group_vars/template/secrets.yml group_vars/[stack_name]/secrets.yml
ansible-vault encrypt --encrypt-vault-id [stack_name] group_vars/[stack_name]/secrets.yml
The encrypted secrets.yml
can now safely be committed.
The .vault/vault_pass.txt.[stack_name]
file is excluded from the repo using the .vault*
pattern in .gitignore
.
To use use an existing encrypted group_vars/[stack_name]/secrets.yml
:
.vault/vault_pass.txt.[stack_name]
file to this repo and use a text editor to add the vault password to this file.We use an SSH public-private key pair to sign the host keys of all the machines in a cluster.
This way users only need the public key of the CA in their ~.ssh/known_hosts
file
and will not get bothered by messages like this:
The authenticity of host '....' can't be established.
ED25519 key fingerprint is ....
Are you sure you want to continue connecting (yes/no)?
[stack_name]-ca
A different CA key file must be specified using the ssh_host_signer_ca_private_key
variable defined in group_vars/[stack_name]/vars.yml
.pub
ssh_host_signer_ca_private_key_pass
variable defined in group_vars/[stack_name]/secrets.yml
,
which must be encrypted with ansible-vault
.~.ssh/known_hosts
like this:
@cert-authority [names of the hosts for which the cert is valid] [content of the CA public key]
E.g.:
@cert-authority reception*,*talos,*tl-* ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDWNAF....VMZpZ5b9+5GA3O8w== UMCG HPC Development CA
ed25519
algorithm and encryption after that:
ssh-keygen -t ed25519 -a 101 -f ssh-host-ca/[stack_name]-ca -C "CA key for [stack_name]"
ansible-vault encrypt --encrypt-vault-id [stack_name] ssh-host-ca/[stack_name]-ca
docker
group.
Otherwise you will get this error:
ERRO[0000] failed to dial gRPC: cannot connect to the Docker daemon.
Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect:
permission denied
context canceled
cd promtools
./build.sh
Execute:
mkdir -p files/[stack_name]
dd if=/dev/urandom bs=1 count=1024 > files/[stack_name]/munge.key
ansible-vault encrypt --encrypt-vault-id [stack_name] files/[stack_name]/munge.key
The encrypted files/[stack_name]/munge.key
can now be committed safely.
If you do not configure any LDAP domains using the ldap_domains
variable (see ldap_server role for details) in group_vars/[stack_name]/vars.yml
,
then the machines for the [stack_name] stack will use local accounts created on each machine and this step can be skipped.
If you configured ldap_domains
in group_vars/[stack_name]/vars.yml
and all LDAP domains have create_ldap: false
,
then this stack will/must use an external LDAP, that was configured & hosted elsewhere, and this step can be skipped.
If you configured one or more LDAP domains with create_ldap: true
; E.g.:
ldap_domains:
stack:
create_ldap: true
.....
other_domain:
some_config_option: anothervalue
create_ldap: true
.....
Then this stack will create and run its own LDAP server. You will need to create:
root
account.readonly
account with a correct dn, password and corresponding hash.admin
account with a correct dn, password and corresponding hash.Create key and CA certificate with one command
openssl req -x509 -nodes -days 1825 -newkey rsa:4096 -keyout files/[stack_name]/ldap.key -out files/[stack_name]/ldap.crt
where you must correctly provide the following values
Country Name (2 letter code) [XX]:NL
State or Province Name (full name) []:Groningen
Locality Name (eg, city) [Default City]:Groningen
Organization Name (eg, company) [Default Company Ltd]:UMCG
Organizational Unit Name (eg, section) []:GCC
Common Name (eg, your name or your server's hostname) []:ladap
Email Address []:hpc.helpdesk@umcg.nl
Note that the Common Name
must be the address of the ldap server. Based on the type of the network access to the machine:
fd-dai
or ladap
),if it is going to be used externally then provide fqdn (like ladap.westeurope.cloudapp.azure.com
).
openssl dhparam -out files/[stack_name]/dhparam.pem 4096
ansible-vault encrypt --encrypt-vault-id [stack_name] files/[stack_name]/ldap.key
ansible-vault encrypt --encrypt-vault-id [stack_name] files/[stack_name]/ldap.crt
ansible-vault encrypt --encrypt-vault-id [stack_name] files/[stack_name]/dhparam.pem
The encrypted files in files/[stack_name]/
can now be committed safely.
When an OpenLDAP server is created, you will need passwords and corresponding hashes for the LDAP root account
as well as for functional accounts for at least one LDAP domain. Therefore the minimal setup in group_vars/[stack_name]/secrets.yml
is something like this:
openldap_root_pw: ''
openldap_root_hash: ''
ldap_credentials:
stack:
readonly:
dn: 'cn=readonly,dc={{ use stack_name here }},dc=local'
pw: ''
hash: ''
admin:
dn: 'cn={{ use stack_prefix here }}-admin,dc={{ use stack_name here }},dc=local'
pw: ''
hash: ''
In this example the LDAP domain named stack
is used for users & groups, that were created for and are used only on this stack of infra.
You may have additional LDAP domains serving as other sources for users and groups.
The pw
values may have been already generated with the generate_secrets.py
script in step 3.
If you added additional LDAP domains later you can, decrypt the group_vars/[stack_name]/secrets.yml
with ansible-vault
,
rerun the generate_secrets.py
script to generate additional password values and re-encrypt secret.yml
with ansible-vault
.
For each pw
you will need to generate a corresponding hash. You cannot use generate_secrets.py
for that,
because it requires the slappasswd
. Therefore, you have to login on the OpenLDAP servers and use:
/usr/local/openldap/sbin/slappasswd \
-o module-path='/usr/local/openldap/libexec/openldap' \
-o module-load='argon2' -h '{ARGON2}' \
-s 'pw_value'
The result is a string with 6 $
separated values like this:
'{ARGON2}$argon2id$v=19$m=65536,t=2,p=1$7+plp......nDs5J!dSpg$ywJt/ug9j.........qKcdfsgQwEI'
For the record:
{ARGON2}
: identifies which hashing schema was used.argon2id
: lists which Argon 2 algorithm was used.v=19
: version of the Argon 2 algorithm.m=65536,t=2,p=1
: lists values used for arguments for the Argon 2 algorithm.7+plp......nDs5J!dSpg
: The base64 encoded radom salt that was added by slappasswd
Use the entire strings as the hash
values in group_vars/[stack_name]/secrets.yml
.
There are two wrapper playbooks:
openstack.yml
:
static_inventories/*.yaml
parsed with our custom inventory plugin inventory_plugins/yaml_with_jumphost.py
cluster.yml
:
openstack.yml
playbook.static_inventories/*.yaml
parsed with our custom inventory plugin inventory_plugins/yaml_with_jumphost.py
The wrapper playbooks execute several roles in the right order to create the complete stack
.
Playbooks from the single_role_playbooks/
or single_group_playbooks/
sub directories can be used to
(re)deploy individual roles or all roles for only a certain type of machine (inventory group), respectively.
These shorter subset playbooks can save a lot of time during development, testing or regular maintenance.
*-openrc.sh
file in the root of the repo.#
# Activate Python virtual env created in step 0.
#
source openstacksdk-${openstacksdk_major_version:-3}.venv/bin/activate
#
# Initialize the OpenstackSDK
#
source ./[Application_Credential_Name]-openrc.sh
#
# Configure this repo for deployment of a specific stack.
#
source ./lor-init
lor-config [stack_prefix]
ansible-playbook openstack.yml
Without local admin accounts we'll need to use
root
account for direct loginsudo su
to become the root user.In our case the CentOS cloud image comes with a default centos
user.
Note that:
~/.ssh/authorized_keys
from an external storage system.centos
account will become useless after the first steps of the playbook have been deployed,
because its home dir with ~/.ssh/authorized_keys
is located in /home,
which will vanish when we mount homes from shared storage.
Changing the location of the default centos
account is not trivial and can result in a situation where you lock yourself out.Therefore the first step is to create additional local admin accounts:
sudo su
to the root user.Without signed host keys, SSH host key checking must be disabled for this first step. The next step is to deploy the signed host keys. Once these first two steps have been deployed, the rest of the steps can be deployed with a local admin account and SSH host key checking enabled, which is the default.
In order to reach machines behind the jumphost you will need to configure your SSH client.
The templates for the documentation are located in this repo at:
roles/online_docs/templates/mkdocs/docs/
Deployed docs can currently be found at:
http://docs.gcc.rug.nl/
Once configured correctly you should be able to do a multi-hop SSH via a jumphost to a destination server using aliases like this:
ssh user@jumphost+destination
export JUMPHOST_USER='user_on_jumphost'
ssh user_on_destination@jumphost+destination
export AI_PROXY='reception'
export ANSIBLE_INVENTORY='static_inventories/talos_cluster.yml'
export ANSIBLE_VAULT_IDENTITY_LIST='all@.vault/vault_pass.txt.all, talos@.vault/vault_pass.txt.talos_cluster'
This can also be accomplished with less typing by sourcing an initialisation file, which provides the lor-config
function
to configure these environment variables for a specific cluster/site:
. ./lor-init
lor-config tl
#
# CentOS 7.x default_cloud_image_user = centos
# Rocky 9.x default_cloud_image_user = cloud-user
#
default_cloud_image_user='centos|cloud-user'
lor_admin_user='your_admin_account'
init.yml
and
configure other stuff on the jumphost (contains amongst others the settings required to access the other machines behind the jumphost)
with cluster.yml
:
ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u "${default_cloud_image_user}" -l 'jumphost' single_group_playbooks/init.yml
ansible-playbook -u "${lor_admin_user}" -l 'jumphost' cluster.yml
init.yml
you must (temporarily) set JUMPHOST_USER
for access to the jumphost to your local admin account,
because the ${default_cloud_image_user}
user will no longer be able to login to the jumphost:
export JUMPHOST_USER="${lor_admin_user}" # Requires SSH client config as per end user documentation: see above.
ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u "${default_cloud_image_user}" -l '!jumphost' single_group_playbooks/init.yml
unset JUMPHOST_USER
ansible-playbook -u "${lor_admin_user}" -l '!jumphost' cluster.yml
ansible-playbook -u "${lor_admin_user}" single_role_playbooks/rsyslog_client.yml
See the end user documentation, that was generated with the online_docs
role for instructions how to submit a job to test the cluster.