WIP: Containerise prometheus

[sjpb]

• Bumps prometheus from 2.27.0 to 2.48.1
• Adds new prometheus ansible role to run prometheus containerised using podman/systemd:
  • Supports previous cloudalchemy.prometheus role config options in use by the appliance.
  • Provides install.yml, runtime.yml playbooks for separate image-build / runtime-configuration use for speed.
• Extends podman role with service templates from azimuth-images, to start convergence of codebases.

WIP, with problems - see https://wiki.stackhpc.com/doc/containerised-prometheus-oLfxe5Es6K for notes

TODO:

• [x] Test fat image build
• [ ] Test upgrade on Azimuth

[sjpb]

Tested an update (via branch switch and running site.yml again) from bfa719f works, i.e. monitoring from before update is visible afterwards.

[sjpb]

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/7411834185 - openhpc-240104-1602-262d12b5

[sjpb]

New image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/7423991616

[sjpb]

Checked at 3cff73b that upgrading an azimuth-deployed slurm to this works on azimuth from azimuth-config d2e6ee2 / v0.3.2:

• Deployed azimuth at azimuth-config v0.3.2
• Deployed slurm with hpctests
• Checked prometheus: clicked through from jobs list to prom data for a job
• Updated caas slurm definition to 3cff73b and corresponding image and re-provision azimuth
• Patch cluster
• Checked prometheus: checked could still click through from slurm jobs dashboard and see data for job run before patch.