build on centos-stream nova_wait_for_compute_service container failure (stage5)

[gprocunier]

Fresh build on CentOS stream today using : CentOS-Stream-GenericCloud-9-20230508.0.x86_64.qcow2

Stage5 is pretty late in the deploy so I am curious what might be up.

2023-05-16 09:36:59.049850 | 52540099-d186-cfdd-423f-0000000055d9 |       TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5
2023-05-16 09:47:06.112097 |                                      |    WARNING | ERROR: Can't run container nova_wait_for_compute_service
stderr: time="2023-05-16T13:37:01Z" level=info msg="podman filtering at log level info"
time="2023-05-16T13:37:01Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
time="2023-05-16T13:37:01Z" level=info msg="Setting parallel job count to 25"
time="2023-05-16T13:37:01Z" level=info msg="Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host"
time="2023-05-16T13:37:01Z" level=info msg="User mount overriding libpod mount at \"/etc/hosts\""
time="2023-05-16T13:37:01Z" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-9d63dcda30c82cd018bd00cdb5ec65bb6c01ffed46ee535f09c71b486a06645c.scope"
time="2023-05-16T13:37:01Z" level=info msg="Got Conmon PID as 113598"
+ sudo -E kolla_set_configs
time="2023-05-16T13:37:01Z" level=info msg="Received shutdown.Stop(), terminating!" PID=113578
sudo: unable to send audit message: Operation not permitted
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Deleting /etc/libvirt/virtlogd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtlogd.conf to /etc/libvirt/virtlogd.conf
INFO:__main__:Deleting /etc/libvirt/virtnodedevd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtnodedevd.conf to /etc/libvirt/virtnodedevd.conf
INFO:__main__:Deleting /etc/libvirt/virtproxyd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtproxyd.conf to /etc/libvirt/virtproxyd.conf
INFO:__main__:Deleting /etc/libvirt/virtqemud.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtqemud.conf to /etc/libvirt/virtqemud.conf
INFO:__main__:Deleting /etc/libvirt/virtsecretd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtsecretd.conf to /etc/libvirt/virtsecretd.conf
INFO:__main__:Deleting /etc/libvirt/virtstoraged.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtstoraged.conf to /etc/libvirt/virtstoraged.conf
INFO:__main__:Deleting /etc/nova/migration/authorized_keys
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys
INFO:__main__:Deleting /etc/nova/migration/identity
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity
INFO:__main__:Creating directory /etc/nova/provider_config
INFO:__main__:Deleting /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/policy.yaml to /etc/nova/policy.yaml
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml
INFO:__main__:Deleting /etc/sasl2/libvirt.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf
INFO:__main__:Deleting /etc/ssh/ssh_config
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/ssh_config to /etc/ssh/ssh_config
INFO:__main__:Deleting /etc/ssh/sshd_config
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config
INFO:__main__:Deleting /etc/login.defs
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/login.defs to /etc/login.defs
INFO:__main__:Deleting /var/lib/nova/.ssh/config
INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/log/nova
INFO:__main__:Setting permission for /var/log/nova/nova-manage.log
INFO:__main__:Setting permission for /var/log/nova/nova-scheduler.log
INFO:__main__:Setting permission for /var/log/nova/nova-conductor.log
INFO:__main__:Setting permission for /var/log/nova/nova-novncproxy.log
INFO:__main__:Setting permission for /var/log/nova/nova-metadata-api.log
INFO:__main__:Setting permission for /var/log/nova/nova-api.log
++ cat /run_command
+ CMD='/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py'\'''
+ umask 0022
+ exec /container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_compute_service.py
+ command -v python3
+ python3 /container-config-scripts/nova_wait_for_compute_service.py
2023-05-16 09:47:06.112659 | 52540099-d186-cfdd-423f-0000000055d9 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | rdo-centos9 | error={"changed": false, "msg": "Failed containers: nova_wait_for_compute_service"}
2023-05-16 09:47:06.112900 | 52540099-d186-cfdd-423f-0000000055d9 |     TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | rdo-centos9 | 0:27:48.100990 | 607.06s

[gprocunier]

local-overrides.conf

standalone_host: 192.168.124.29
public_api: 192.168.124.29
rhos_release: 17.0
hostname: rdo-centos9
clouddomain: home.lan
local_cloudname: rhos-17
openshift_password: openshift
ceph_devices:
  - /dev/sda

inventory.yaml

all:
  hosts:
    standalone:
      ansible_host: 192.168.124.29
      ansible_user: root

[gprocunier]

standalone-install-20230516134706.tar.gz

[EmilienM]

it's due to https://bugzilla.redhat.com/show_bug.cgi?id=1498621

[EmilienM]

workaround:

extra_heat_params:
  NovaComputeStartupDelay: 0