Slurm README

Slurm on IBM Cloud enables customers to use Terraform as an automation tool to easily and quickly deploy Slurm clusters. The automation has been validated and tested for Slurm 21.08.5-2ubuntu1 Advanced Edition. For more details you can refer to our documentation

Support

Depending on the area where the issue is encountered use the following channels for getting help with the solution:

1. If the issue is with automation code, you can create an issue on this git repository
2. If the issue is with IBM Cloud infrastructure you can create a cloud support case here
3. If the issue is with Slurm software, you can work with SchedMD corporation and get help here

Deployment with Schematics CLI on IBM Cloud

Initial configuration:

$ cp sample/configs/hpc_workspace_config.json config.json
$ ibmcloud iam api-key-create my-api-key --file ~/.ibm-api-key.json -d "my api key"
$ cat ~/.ibm-api-key.json | jq -r ."apikey"
# copy your apikey
$ vim config.json
# paste your apikey

You also need to generate github token if you use private Github repository.

Deployment:

$ ibmcloud schematics workspace new -f config.json --github-token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ ibmcloud schematics workspace list
Name               ID                                            Description   Status     Frozen
hpcc-slurm-test       us-east.workspace.hpcc-slurm-test.7cbc3f6b                     INACTIVE   False

OK
$ ibmcloud schematics plan --id us-east.workspace.hpcc-slurm-test.7cbc3f6b

Activity ID b0a909030f071f51d6ceb48b62ee1671

OK
$ ibmcloud schematics apply --id us-east.workspace.hpcc-slurm-test.7cbc3f6b
Do you really want to perform this action? [y/N]> y

Activity ID b0a909030f071f51d6ceb48b62ee1672

OK
$ ibmcloud schematics logs --id us-east.workspace.hpcc-slurm-test.7cbc3f6b
...
 2023/04/26 08:20:37 Terraform apply | Apply complete! Resources: 34 added, 0 changed, 0 destroyed.
 2023/04/26 08:20:37 Terraform apply | 
 2023/04/26 08:20:37 Terraform apply | Outputs:
 2023/04/26 08:20:37 Terraform apply | 
 2023/04/26 08:20:37 Terraform apply | nfs_ssh_command = "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -J root@158.177.2.7 root@10.243.0.36"
 2023/04/26 08:20:37 Terraform apply | region_name = "eu-de"
 2023/04/26 08:20:37 Terraform apply | ssh_command = "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -J root@158.177.2.7  root@10.243.0.37"
 2023/04/26 08:20:37 Terraform apply | vpc_name = "marvel-slurm-vpc --  - r010-5d7c29b4-5585-49bc-bfde-ab145d9f104b"
 2023/04/26 08:20:37 Command finished successfully.

OK
$ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -J root@158.177.2.7  root@10.243.0.37"

$ ibmcloud schematics destroy --id us-east.workspace.hpcc-slurm-test.7cbc3f6b
Do you really want to perform this action? [y/N]> y

Activity ID b0a909030f071f51d6ceb48b62ee1673

OK

Deployment with Schematics UI on IBM Cloud

1. Go to https://cloud.ibm.com/schematics, select Workspaces and then select create workspace using Schematics.
2. In the Specify template section:
   • Provide your GitHub, GitLab or Bitbucket repository URL where your Terraform files resides.
   • If you are using a private GitHub repository, provide your personal GitHub access token that you set up in Setting up the IBM Cloud Schematics prerequisites.
   • Select the terraform version, the version of the Terraform engine that's used in the Schematics workspace, and then click Next.
3. In the workspace details section:
   • Specify the Name for your Schematics workspace
   • Define any Tags that you want to associate with the resources provisioned through the offering. The tags can later be used to query the resources in the IBM Cloud console.
   • Select a Resource group
   • Select a Location. Location determines where workspace actions will be executed
   • Provide a Description (optional) of the Schematics workspace.
   • Click on Next and Then click Create. The Schematics workspace is created with the name you specified.
4. Go to Schematic Workspace Settings, under variable section, click on "burger icons" to update the following parameters:
   • ssh_key_name with your ibm cloud SSH key name such as "slurm-ssh-key" created in a specific region in IBM Cloud
   • api_key with the api key value and mark it as sensitive to hide the API key in the IBM Cloud Console
   • Update cluster_prefix value to the specific cluster prefix for your Slurm cluster
   • Update cluster_id the ID of the cluster used by Slurm for configuration of resources
   • Update the worker_node_count as per your requirement
5. Click on "Generate Plan" and ensure there are no errors and fix the errors if there are any.
6. After "Generate Plan" gives no errors, click on "Apply Plan" to create resources.
7. When you click on "Apply plan" the post provisioning scripts run and the Slurm workload manager get installed.
8. Click on Jobs then Activity and Show more to view the resource creation progress.
9. Click on "Show more" if the "Apply Plan" activity is successful and copy the output SSH command to your laptop terminal to SSH to management node via a jump host public ip to SSH one of the nodes.
10. Also use this jump host public ip and change the IP address of the node you want to access via the jump host to access specific hosts.

Storage Node and NFS Setup

The storage node is configured as an NFS server and the data volume is mounted to the /data directory which is exported to share with Slurm management nodes.

Steps to validate Cluster setups

How to login cluster nodes:

Storage Node

- On a local machine, ssh [-i <identity_file>] -J root@<floating_ip_of_bastion_node> ubuntu@<private_ip_of_storage_node>
  Example: ssh -J root@52.117.4.140  root@10.243.0.36 (or) ssh -i id_rsa -J root@52.117.4.140  root@10.243.0.36

Slurm Management / Slurm Worker / Scale Storage Node

- On a local machine, ssh [-i <identity_file>] -J root@<floating_ip_of_bastion_node> ubuntu@<private_ip_for_the_node>
  Example:ssh -J root@52.117.4.140  ubuntu@10.243.0.37 (or) ssh -i id_rsa -J root@52.117.4.140  ubuntu@10.243.0.37

1. To validate the NFS storage is setup and exported correctly

• Login to the storage node using SSH (ssh -J root@ root@)
• The command below shows that the data volume, /dev/vdd, is mounted to /data on the storage node.

  # df -k | grep data
  /dev/vdd       104806400   33008 104773392   1% /data`

• The command below shows that /data is exported as a NFS shared directory.

# exportfs -v
/data         <private_ip_of_storage_node>/27(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

• At the NFS client end, the Slurm management/worker nodes in this case, we mount the /data directory in NFS server to the local directory, /data.

  # df -k | grep data
  10.243.0.36:/data  100G   32M  100G   1% /mnt/data

  The command above shows that the local directory, /data, is mounted to the remote /data directory on the NFS server, 10.240.128.26.

2. Steps to validate the cluster status

• Login to the management as shown in the ssh_command output.

  # ssh -J root@<floating_ip_of_bastion_node> ubuntu@<private_ip_of_management_node>

• Check the status of the munge and slurm management daemon.

  # systemctl status munge
  # systemctl status slurmctld

• The command below show that the nfs is mounted correctly /data.

  # showmount -e <private_ip_of_storage_node>

3. Steps to validate the cluster status

• Login to the worker as shown in the ssh_command output.

  # ssh -J root@<floating_ip_of_bastion_node> ubuntu@<private_ip_of_worker_node>

• Check the status of the munge and slurm node daemon.

  # systemctl status munge
  # systemctl status slurmd

• The command below show that the nfs is mounted correctly /data.

  # showmount -e <private_ip_of_storage_node>

Testing

• The command below reports the state of partitions and nodes managed by Slurm.

  # sinfo -N -l

• The command below reports the state of jobs or job steps. By default, it reports the running jobs in priority order and then the pending jobs in priority order.

  # squeue

• The below command is used to allocate resources for a job in real time. Typically this is used to allocate resources and spawn a shell. The shell is then used to execute srun commands to launch parallel tasks.

  # salloc

• The below commands can be used to report more detailed information about nodes, partitions, jobs, job steps, and configuration.

  # scontrol show jobs
  # scontrol show nodes

• The below commannd is used to cancel a pending or running job or job step. It can also be used to send an arbitrary signal to all processes associated with a running job or job step.

  # scancel <job_id>

• To use a specific number of nodes (one node as N1) and execute the command "hostname" on the worker nodes.

  # srun -N1 -l hostname
  0: hpcc-slurm-worker-0

Stop ufw firewall

• Stop ufw firewall on all nodes (management and worker) if it doesn't work and restart slurmctld/slurmd or munge daemons.

  # systemctl status ufw
  # systemctl stop ufw

  Slurm-wlm

• The command below shows that which slurm version is installed.

  # dpkg -l | grep slurm
  ii  slurm-client                   21.08.5-2ubuntu1            amd64        SLURM client side commands
  ii  slurm-wlm                      21.08.5-2ubuntu1            amd64        Simple Linux Utility for Resource Management
  ii  slurm-wlm-basic-plugins        21.08.5-2ubuntu1            amd64        SLURM basic plugins
  ii  slurm-wlm-doc                  21.08.5-2ubuntu1            all          SLURM documentation
  ii  slurmctld                      21.08.5-2ubuntu1            amd64        SLURM central management daemon
  ii  slurmd                         21.08.5-2ubuntu1            amd64        SLURM compute node daemon

Authentication issues

Issues with munge => munge daemon or munge key not same on all nodes

Steps to validate spectrum scale integration

How to login Scale Storage nodes and Access Scale GUI:

Scale Storage Node

- On a local machine, ssh [-i <identity_file>] -L 22443:localhost:443 -J root@<floating_ip_of_bastion_node> ubuntu@<private_ip_of_scale_storage_node>
  Example: ssh -L 22443:localhost:443 -J root@52.117.4.140  root@10.243.0.39 (or) ssh -i id_rsa -L 22443:localhost:443 -J root@52.117.4.140  root@10.243.0.39

Access Scale Storage GUI

- On local machine, open a local browser window and go to this URL:  https://localhost:22443

• The below command shows the gpfs cluster setup on scale storage node.

  # /usr/lpp/mmfs/bin/mmlscluster

• The below command shows file system mounted on number of nodes

  # /usr/lpp/mmfs/bin/mmlsmount all

• The below command shows the fileserver details. This command can be used to validate file block size(Inode size in bytes).

  #   /usr/lpp/mmfs/bin/mmlsfs all -i

• Create a file on mountpoint path(e.g /gpfs/fs1) and verify on other nodes that the file can be accessed.

Requirements

Providers

Modules

Resources

Inputs

[
  "10.241.0.0/18"
]

[
  "10.241.4.0/28"
]

[
  "10.241.0.0/22"
]

Outputs