virt-customize guestfs_launch error on CentOS 8 Train seed-deploy

[oneswig]

TASK [Ensure the overcloud host image has bogus name server entries removed] ****************************************************************************************************
fatal: [seed]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "bifrost_deploy", "bash", "-c", " export LIBGUESTFS_BACKEND=direct && ansible localhost --connection local --become -m command -a \"virt-customize -a /httpboot/deployment_image.qcow2 --edit \\\"/etc/resolv.conf:s/^nameserver .*\\..*\\..*\\..*\\$//\\\"\""], "delta": "0:00:07.395318", "end": "2020-07-20 16:57:56.264315", "msg": "non-zero return code", "rc": 2, "start": "2020-07-20 16:57:48.868997", "stderr": "[WARNING]: No inventory was parsed, only implicit localhost is available", "stderr_lines": ["[WARNING]: No inventory was parsed, only implicit localhost is available"], "stdout": "localhost | FAILED | rc=1 >>\n[   0.0] Examining the guest ...virt-customize: error: libguestfs error: guestfs_launch failed.\nThis usually means the libguestfs appliance failed to start or crashed.\nDo:\n  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand run the command again.  For further information, read:\n  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can also run 'libguestfs-test-tool' and post the *complete* output\ninto a bug report or message to the libguestfs mailing list.\n\nIf reporting bugs, run virt-customize with debugging enabled and include \nthe complete output:\n\n  virt-customize -v -x [...]non-zero return code", "stdout_lines": ["localhost | FAILED | rc=1 >>", "[   0.0] Examining the guest ...virt-customize: error: libguestfs error: guestfs_launch failed.", "This usually means the libguestfs appliance failed to start or crashed.", "Do:", "  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1", "and run the command again.  For further information, read:", "  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs", "You can also run 'libguestfs-test-tool' and post the *complete* output", "into a bug report or message to the libguestfs mailing list.", "", "If reporting bugs, run virt-customize with debugging enabled and include ", "the complete output:", "", "  virt-customize -v -x [...]non-zero return code"]}

Running virt-customize with more debug in the Bifrost container, I see this output:

qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172
qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.  

A possible root cause appears to be the Bifrost container is now built on CentOS 8.2 but the Seed VM is based on a CentOS 8.1 image.

[markgoddard]

Seems like this will affect kayobe generally, all the way back to Train.

[markgoddard]

Kayobe story: https://storyboard.openstack.org/#!/story/2007942