Add support for RockyLinux9

sjpb commented 5 months ago

Make the appliance compatible with RockyLinux 9-based images.

Note that the CI and CaaS environments will continue to use RL8 at present. CI is only carried out using RL9 if a PR branch name starts with rl9 or RL9 is selected when running CI workflows manually.

Additional notes:

This also removes the Packer template for "environment-specific" compute image builds. No deployments are using this functionality and it has limitations which would be better addressed by making the fatimage usable on boot via cloud-init.

Podman commands in systemd units are changed to use --cgroup-manager=cgroupfs. This was demonstrated to be the default in RL8; for RL9 the default is systemd which leads to log warnings like:

The cgroupv2 manager is set to systemd but there is no systemd user session available"
For using systemd, you may need to login using an user session"
Alternatively, you can enable lingering with: `loginctl enable-linger 1001` (possibly as root)"
Falling back to --cgroup-manager=cgroupfs"
"unlinkat /run/podman/libpod/tmp: permission denied"

However enabling user-lingering leads to mysql failing to start at all, with the relevant error probably being:

user@1001.service: Got notification message from PID 5388, but reception only permitted for main PID 4270

Replaces #323

sjpb commented 5 months ago

FAILED Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/7643335283

Edit: had wrong image type in glance

sjpb commented 5 months ago

Tests at 0982f41 on a "local" cluster:

deploy OK
reboot control: services OK
hpctests OK
OOD shell: OK
slurm jobs: OK
hpl timehistory: OK
slurm exporter: OK
ondemand exporter: some missing data, see below
OOD jupyter OK
OOD desktop: failed, no dbus-launch command

ondemand exporter:

[rocky@rl9-login-0 ~]$ systemctl status ondemand_exporter.service
Jan 24 16:26:24 rl9-login-0.rl9.invalid ondemand_exporter[36589]: ts=2024-01-24T16:26:24.868Z caller=collector.go:171 level=error msg="Error collecting apache information" err="Get \"http://localhost:81/server-status\":>

[rocky@rl9-login-0 ~]$ cat /usr/lib/systemd/system/ondemand_exporter.service
Environment="APACHE_STATUS_URL=http://localhost:81/server-status"

[rocky@rl9-login-0 ~]$ curl localhost:9301/metrics
# shows this is working at least

sjpb commented 5 months ago

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/7653662893

Edit: currently failing due to CVMFS repo 503-ing Edit: repo appears up, retrying

sjpb commented 5 months ago

Checked locally that e5608d9 works on both a) a cluster with existing non-system users b) fresh image

sjpb commented 4 months ago

NB: Currently CI doens't get past the os-manila-mount install task b/c the rpm-reef URL at https://download.ceph.com/ has been broken/renamed.

Edit: see https://tracker.ceph.com/issues/64718

sjpb commented 4 months ago

Repos fixed, lets try again

sjpb commented 4 months ago

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/8186093673