Should we make magic_castle scale dynamically via slurm elastic scheduling?

[consideRatio]

Probably no. I think it would be pushing the scope of the magic castle projects ambition significantly. It would add a lot of complexity specific for the cloud.

Reasons:

• Installation of python packages is not done using puppet currently, and packages are installed on each node on separate local volumes. Due to this, those packages would quickly get out of sync if new nodes are added/removed. Configuration as code is required systematically. Related: https://github.com/ComputeCanada/magic_castle/tree/main/docs#106-add-packages-to-jupyter-default-python-kernel
• terraform_data.yaml generated by terraform apply feed information to puppetserver about the instances in the cluster, but using elastic additions we would have another system that would need to feed puppetserver with state. There would need to be one source of truth.

[consideRatio]

What needs to happen whenever a node is added?

• Public SSH keys configured for a centos user
• Mount storage?
• Puppet agent installed
• Puppet agent sync with Puppet server

How can a node be added

• An admin manually let terraform apply a changed configuration that declares a higher/lower number of nodes.
• Slurm runs a script that manually scale up/down a node pool

[consideRatio]

Meeting notes with Félix 2021-05-21

Strategy idea: to let a slurm-resume / slurm-suspend script call terraform apply.

Notes

• Terraform cloud / github dispatchable workflow could run change the configuration in a predefined way, and then run terraform apply.
• Can terraform manage power state? Maybe, we probably require this as having a decoupled state managed by slurm that conflicts with terraform would be problematic.
• Slurm would need to say exactly what node to suspend, not just decrease one.
• One off configuration of nodes with clush would not be sensible if we have multiple nodes in different power states, but the default configuration doesn't rely on this - only puppet.
• Python packages should probably be installed in a shared storage space. Is there other state that should transition to a shared storage space? No probably not actually.
• Management node stores user information (FreeIPA) in its instance volume.
• Terraform Cloud can keep track and manage the state in a centralized place, which a github repo doesn't do perfectly - maybe?
  • It is possible to configure a centralized location for state, but this would require access to the cloud providers location for storing that state.

Action points

• [ ] Explore if terraform can manage VM power states
• [ ] Explore how to run terrafrom apply from the slurm suspend/resume script
  • [ ] Use Terraform Cloud, or perhaps use a dispatchable GitHub Workflow running a magic-castle defined GitHub Action with version control?
  • [ ] Consider the complexity if Slurm wants to resume one node, then another, then another in short succession. Would the system work well if the goal changes? Related to this is the concepts of edge triggered vs level triggered, should we act whenever something change (edge triggered), or should we continuously act to arrive at a certain state and keep working as we observe this desired state change (level triggered)?
• [ ] Trial Terraform Cloud / Learn practices of managing terraform state