[BUG] AL2023 with systemd 252 cause docker not able to run systemd on cgroupv2

[peterzhuamazon]

In old AL2 we have systemd 219 on the host with selinux disabled, which allows to use cgroupv1 for systemd to run in docker.

In new AL2023 we have systemd 252 on the host with selinux enabled by default, with cgroupv2 that cause these errors to show:

Failed to connect to bus: No such file or directory

...

$ docker run --rm -it --entrypoint=/usr/sbin/init -u root --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro opensearchstaging/ci-runner:ci-runner-rockylinux8-systemd-base-integtest-v3
SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.33:  No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

There are two ways to resolve this:

1. Revert back to the old cgroupv1: https://github.com/moby/moby/issues/42275#issuecomment-1115041405
2. Set cgroup rw and use cgroupns=host: https://github.com/izer-xyz/3cx-docker/issues/4

Additionally on AL2023, simply alter the /etc/selinux/config or setenforce will not change the behavior despite selinux showing sestatus as disabled on AL2023.

Use this, which needs to be set on the lib/compute/agent-nodes.ts in opensearch-ci repo.

sudo grubby --update-kernel ALL --args selinux=0

Then use this:

$  docker run --rm -it --entrypoint=/usr/lib/systemd/systemd -u root --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:rw --cgroupns=host opensearchstaging/ci-runner:ci-runner-rockylinux8-systemd-base-integtest-v3

Thanks.

[peterzhuamazon]

PRs:

• https://github.com/opensearch-project/opensearch-ci/pull/351
• https://github.com/opensearch-project/opensearch-ci/pull/352
• https://github.com/opensearch-project/opensearch-build/pull/4050
• https://github.com/opensearch-project/opensearch-build/pull/4054

[peterzhuamazon]

More related:

• https://github.com/moby/moby/pull/42043

[peterzhuamazon]

We have resolved this issue with both deb and rpm fixed in IntegTest now.

Thanks.