rhinstaller / kickstart-tests

Kickstart-driven tests for the Anaconda installer
53 stars 34 forks source link

Kickstart tests are broken. #1129

Closed rvykydal closed 2 months ago

rvykydal commented 2 months ago

Kickstart test are broken, first observed in this action: https://github.com/rhinstaller/kickstart-tests/actions/runs/8585781885

On rawhide / daily-iso the tests are failing with:

2024-04-08 10:18:56,420 DEBUG: Failed to run subprocess: '['/opt/kstest/kickstart-tests/scripts/launcher/lib/launcher_interface.sh', '-i', '/opt/kstest/data/images/boot.iso', '-k', '1', '-w', '/var/tmp/kstest-reboot-initial-setup-gui.2024_04_08-09_47_17.kh8wpxjr', '-t', '/opt/kstest/kickstart-tests/reboot-initial-setup-gui.sh', 'validate']'
stderr:
cat: /var/tmp/kstest-reboot-initial-setup-gui.2024_04_08-09_47_17.kh8wpxjr/RESULT: No such file or directory
stdout:
*** /root/RESULT does not exist in VM image.

2024-04-08 10:18:56,420 INFO: RESULT:reboot-initial-setup-gui:61e58a19f53b:FAILED:Validation failed with return code 1

On rhelx, they hang after start (maybe probe boot.iso)

INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] ************************************************************************
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] You can connect to this container's libvirt with this connection string:
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] 
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] qemu+tcp://10.88.0.25/session
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] 
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] ************************************************************************
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] + TEST_LOG=/var/tmp/kstest.log
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] + pushd /opt/kstest/kickstart-tests
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] /opt/kstest/kickstart-tests /opt/kstest
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] + set +e
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] + scripts/run_kickstart_tests.sh -x 0 -k 1 -i /opt/kstest/data/images/boot.iso -p rhel8 keyboard
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] + tee /var/tmp/kstest.log
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] starting kickstart tests
INFO:libpermian.plugins.kickstart_test(Thread-5):[runner] Mon Apr  8 13:17:33 UTC 2024

To reproduce, just run

/test-platforms

in some PR comment in this repository

rvykydal commented 2 months ago

Given that for rawhide / daily-iso the tests fail in the same way on Travis and our runners, the environment to investigate would be the container we use for running kstests (quay.io/rhinstaller/kstest-runner:latest) ?

This function may be failing (https://github.com/rhinstaller/kickstart-tests/blob/061ae29e88b12f0fe3e51788eb3db772db3d1d35/functions.sh#L155)

rvykydal commented 2 months ago

Kickstart test are broken, first observed in this action: https://github.com/rhinstaller/kickstart-tests/actions/runs/8585781885

The last container update happened quite close before https://github.com/rhinstaller/kickstart-tests/actions/runs/8582993864/job/23521715932

rvykydal commented 2 months ago

Reproducible locally with the latest container.

rvykydal commented 2 months ago

It seems that the cause is hanging virt-cat in https://github.com/rhinstaller/kickstart-tests/blob/061ae29e88b12f0fe3e51788eb3db772db3d1d35/scripts/probe_boot_iso.sh#L76 (for rhel runs) and virt-copy-out in https://github.com/rhinstaller/kickstart-tests/blob/061ae29e88b12f0fe3e51788eb3db772db3d1d35/functions.sh#L116 (for fedora runs) in the latest kstest-runner container

rvykydal commented 2 months ago

Running virt-copy-out in the container manually with trace messages hangs:

[rvykydal@prnot4 ~/work/git/rvykydal/kickstart-tests.devel (dbg-hang)]$ sudo podman ps
[sudo] password for rvykydal:
CONTAINER ID  IMAGE                                     COMMAND               CREATED             STATUS             PORTS                       NAMES
7e40fc65bc2a  quay.io/rhinstaller/kstest-runner:latest  /kickstart-tests/...  About a minute ago  Up About a minute  127.0.0.1:44911->16509/tcp  mystifying_lehmann
[rvykydal@prnot4 ~/work/git/rvykydal/kickstart-tests.devel (dbg-hang)]$ sudo podman exec -ti mystifying_lehmann /bin/bash
[kstest@7e40fc65bc2a kstest]$ ps
    PID TTY          TIME CMD
    870 pts/1    00:00:00 bash
    889 pts/1    00:00:00 ps
[kstest@7e40fc65bc2a kstest]$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
kstest         1  0.0  0.0   4080  3072 pts/0    Ss+  07:02   0:00 /bin/sh /kickstart-tests/containers/runner/run-kstest
kstest         7  0.4  0.0 1438764 27756 pts/0   Sl+  07:02   0:00 libvirtd --listen
kstest        12  0.0  0.0   4212  3072 pts/0    S+   07:02   0:00 /bin/bash scripts/run_kickstart_tests.sh -x 0 -k 1 -i /opt/kstest/data/images/boot.iso container
kstest        13  0.0  0.0   2476  1280 pts/0    S+   07:02   0:00 tee /var/tmp/kstest.log
kstest       284  0.0  0.0   2480  1536 pts/0    S    07:02   0:00 timeout 0 parallel --no-notice --jobs 8 PYTHONPATH= scripts/launcher/run_one_test.py -i /opt/kstest/data/images/boot.iso -k 1 --append-host-id {} ::: container.sh
kstest       285  0.0  0.0  22072 19804 pts/0    S    07:02   0:00 /usr/bin/perl /usr/bin/parallel --no-notice --jobs 8 PYTHONPATH= scripts/launcher/run_one_test.py -i /opt/kstest/data/images/boot.iso -k 1 --append-host-id {} ::: container.sh
kstest       331  0.9  0.1 158524 56364 pts/0    S    07:02   0:01 /usr/bin/python3 scripts/launcher/run_one_test.py -i /opt/kstest/data/images/boot.iso -k 1 --append-host-id container.sh
kstest       637  0.0  0.0 182444 17792 ?        Sl   07:02   0:00 /usr/sbin/virtlogd --timeout=120
kstest       691  0.0  0.0   4724  3328 pts/0    S    07:03   0:00 /bin/bash /opt/kstest/kickstart-tests/scripts/launcher/lib/launcher_interface.sh -i /opt/kstest/data/images/boot.iso -k 1 -w /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh -t /opt
kstest       697  0.0  0.0   4724  2392 pts/0    S    07:03   0:00 /bin/bash /opt/kstest/kickstart-tests/scripts/launcher/lib/launcher_interface.sh -i /opt/kstest/data/images/boot.iso -k 1 -w /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh -t /opt
kstest       699  0.0  0.0   2860  1664 pts/0    S    07:03   0:00 timeout -k 10s 300s virt-copy-out -a /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img /root/original-ks.cfg /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh
kstest       700  0.0  0.0  38804 20752 pts/0    S    07:03   0:00 /usr/bin/virt-copy-out --ro -i copy-out -a /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img /root/original-ks.cfg /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh
kstest       864 99.5  0.5 1708576 183452 ?      Sl   07:03   0:56 /usr/bin/qemu-kvm -name guest=guestfs-97iluj4wax8scmrj,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/home/kstest/.config/libvirt/qemu/lib/doma
kstest       870  0.0  0.0   4768  3712 pts/1    Ss   07:04   0:00 /bin/bash
kstest       890  0.0  0.0   5960  2816 pts/1    R+   07:04   0:00 ps aux
[kstest@7e40fc65bc2a kstest]$ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
[kstest@7e40fc65bc2a kstest]$ /usr/bin/virt-copy-out --ro -i copy-out -a /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img /root/original-ks.cfg .
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: create: flags = 0, handle = 0x558090eb4260, program = virt-copy-out
libguestfs: trace: set_pgroup true
libguestfs: trace: set_pgroup = 0
libguestfs: trace: add_drive "/var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img" "readonly:true"
libguestfs: creating COW overlay to protect original drive content
libguestfs: trace: disk_format "/var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img"
libguestfs: command: run: qemu-img --help | grep -sqE -- '\binfo\b.*-U\b'
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ -U
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img
libguestfs: parse_json: qemu-img info JSON output:\n{\n    "virtual-size": 10737418240,\n    "filename": "/var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img",\n    "cluster-size": 65536,\n    "format": "qcow2",\n    "actual-size": 120524800,\n    "format-specific": {\n        "type": "qcow2",\n        "data": {\n            "compat": "1.1",\n            "compression-type": "zlib",\n            "lazy-refcounts": false,\n            "refcount-bits": 16,\n            "corrupt": false,\n            "extended-l2": false\n        }\n    },\n    "dirty-flag": false\n}\n\n
libguestfs: trace: disk_format = "qcow2"
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: disk_create "/tmp/libguestfsewy5ZI/overlay1.qcow2" "qcow2" -1 "backingfile:/var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img" "backingformat:qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img,backing_fmt=qcow2
libguestfs: command: run: \ /tmp/libguestfsewy5ZI/overlay1.qcow2
Formatting '/tmp/libguestfsewy5ZI/overlay1.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 backing_file=/var/tmp/kstest-container.2024_04_09-07_02_09.wtesw7gh/disk-a.img backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
libguestfs: trace: disk_create = 0
libguestfs: trace: get_backend_setting "internal_libvirt_imagelabel"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: add_drive = 0
libguestfs: trace: is_config
libguestfs: trace: is_config = 1
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 50, release: 1, extra: fedora=38,release=1.fc38,libvirt, >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "libvirt"
libguestfs: launch: program=virt-copy-out
libguestfs: launch: version=1.50.1fedora=38,release=1.fc38,libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsewy5ZI
libguestfs: launch: umask=0022
libguestfs: launch: euid=1001
libguestfs: libvirt version = 9000000 (9.0.0)
libguestfs: guest random name = guestfs-n9q2zwbpv6enwre0
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///session, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x558090ebef90
libguestfs: qemu version (reported by libvirt) = 7002010 (7.2.10)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: trace: get_backend_setting "force_kvm"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_backend_setting "force_tcg"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: parsing domcapabilities XML
libguestfs: trace: get_backend_setting "internal_libvirt_label"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_backend_setting "internal_libvirt_imagelabel"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_backend_setting "internal_libvirt_norelabel_disks"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: build appliance
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-1001/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-1001/appliance.d
supermin: version: 5.3.4
supermin: rpm: detected RPM version 4.18
supermin: rpm: detected RPM architecture x86_64
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-1001/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-packages-xfs type uncompressed packages
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 212 packages, including dependencies
supermin: build: 33933 files
supermin: build: 9488 files, after matching excludefiles
supermin: build: 9499 files, after adding hostfiles
supermin: build: 9471 files, after removing unreadable files
supermin: build: 9495 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/6.8.4-100.fc38.x86_64/vmlinuz
supermin: kernel: kernel_version 6.8.4-100.fc38.x86_64
supermin: kernel: modpath /lib/modules/6.8.4-100.fc38.x86_64
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-1001/appliance.d.qwp6c8do/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: warning: /usr/libexec/utempter/utempter: Permission denied (ignored)
Some distro files are not public readable, so supermin cannot copy them
into the appliance.  This is a problem with your Linux distro.  Please ask
your distro to stop doing pointless security by obscurity.
You can ignore these warnings.  You *do not* need to use sudo.
supermin: warning: /usr/sbin/unix_update: Permission denied (ignored)
supermin: warning: /var/lib/systemd/random-seed: Permission denied (ignored)
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-1001/appliance.d.qwp6c8do/initrd'
supermin: ext2: wrote 34 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-1001/appliance.d.qwp6c8do to /var/tmp/.guestfs-1001/appliance.d
libguestfs: finished building supermin appliance
libguestfs: trace: disk_create "/tmp/libguestfsewy5ZI/overlay2.qcow2" "qcow2" -1 "backingfile:/var/tmp/.guestfs-1001/appliance.d/root"
libguestfs: trace: disk_format "/var/tmp/.guestfs-1001/appliance.d/root"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ -U
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /var/tmp/.guestfs-1001/appliance.d/root
libguestfs: parse_json: qemu-img info JSON output:\n{\n    "virtual-size": 4294967296,\n    "filename": "/var/tmp/.guestfs-1001/appliance.d/root",\n    "format": "raw",\n    "actual-size": 321646592,\n    "dirty-flag": false\n}\n\n
libguestfs: trace: disk_format = "raw"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-1001/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfsewy5ZI/overlay2.qcow2
Formatting '/tmp/libguestfsewy5ZI/overlay2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 backing_file=/var/tmp/.guestfs-1001/appliance.d/root backing_fmt=raw lazy_refcounts=off refcount_bits=16
libguestfs: trace: disk_create = 0
libguestfs: trace: get_sockdir
libguestfs: trace: get_sockdir = "/tmp"
libguestfs: set_socket_create_context: setsockcreatecon failed: system_u:system_r:svirt_socket_t:s0:c569,c662: Permission denied [you can ignore this message if you are not using SELinux + sVirt]
libguestfs: clear_socket_create_context: setsockcreatecon failed: NULL: Permission denied [you can ignore this message if you are not using SELinux + sVirt]
libguestfs: create libvirt XML
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-n9q2zwbpv6enwre0</name>\n  <memory unit="MiB">1280</memory>\n  <currentMemory unit="MiB">1280</currentMemory>\n  <cpu mode="maximum">\n    <feature policy="disable" name="la57"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-1001/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-1001/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=8b2cb703-5a13-493f-9472-96a1c0a9c7f5 selinux=0 guestfs_verbose=1 TERM=xterm</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <seclabel type="none"/>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <emulator>/usr/bin/qemu-kvm</emulator>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfsewy5ZI/overlay1.qcow2"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfsewy5ZI/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfsN5matv/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsN5matv/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-1001
libguestfs: /var/tmp/.guestfs-1001:
libguestfs: total 0
libguestfs: drwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0  30 Apr  9 07:04 .
libguestfs: drwxrwxrwt. 1 root   root   system_u:object_r:container_file_t:s0 204 Apr  9 07:04 ..
libguestfs: drwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0  32 Apr  9 07:04 appliance.d
libguestfs: -rw-r--r--. 1 kstest kstest system_u:object_r:container_file_t:s0   0 Apr  9 07:04 lock
libguestfs:
libguestfs: /var/tmp/.guestfs-1001/appliance.d:
libguestfs: total 331732
libguestfs: drwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0         32 Apr  9 07:04 .
libguestfs: drwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0         30 Apr  9 07:04 ..
libguestfs: -rw-r--r--. 1 kstest kstest system_u:object_r:container_file_t:s0    3146240 Apr  9 07:04 initrd
libguestfs: -rwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0   14896968 Apr  9 07:04 kernel
libguestfs: -rw-r--r--. 1 kstest kstest system_u:object_r:container_file_t:s0 4294967296 Apr  9 07:04 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfsN5matv
libguestfs: total 0
libguestfs: drwx------. 1 kstest kstest system_u:object_r:container_file_t:s0:c569,c662  50 Apr  9 07:04 .
libguestfs: drwxrwxrwt. 1 root   root   system_u:object_r:container_file_t:s0:c569,c662 154 Apr  9 07:04 ..
libguestfs: srwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0:c569,c662   0 Apr  9 07:04 console.sock
libguestfs: srwxr-xr-x. 1 kstest kstest system_u:object_r:container_file_t:s0:c569,c662   0 Apr  9 07:04 guestfsd.sock
libguestfs: launch libvirt guest
\x1bc\x1b[?7l\x1b[2J\x1b[0mSeaBIOS (version 1.16.2-1.fc38)
Machine UUID 37f19a6c-5366-4fec-be1e-567cf5978f55
Booting from ROM..\x1bc\x1b[?7l\x1b[2J
rvykydal commented 2 months ago

PRs to improve investigating of similar issues: https://github.com/rhinstaller/kickstart-tests/pull/1130 https://github.com/rhinstaller/kickstart-tests/pull/1131

rvykydal commented 2 months ago

It seems that the cause is hanging virt-cat in

https://github.com/rhinstaller/kickstart-tests/blob/061ae29e88b12f0fe3e51788eb3db772db3d1d35/scripts/probe_boot_iso.sh#L76 (for rhel runs)

A reproducer in kickstart-tests checkout, assuming you have rhel9 boot.iso in data/images/boot.iso (enters the container and runs the script which hangs vith virt-cat):

[rvykydal@prnot4 ~/work/git/rvykydal/kickstart-tests.devel (upstream-master)]$ sudo containers/runner/launch -c --platform rhel9 keyboard
+ /usr/bin/podman run -it --rm --device=/dev/kvm --publish 127.0.0.1::16509 --env SCENARIO=unknown --env DRY_RUN= --env KSTESTS_TEST=keyboard --env TESTTYPE= --env SKIP_TESTTYPES= --env KSTESTS_RUN_TIMEOUT= --env TEST_JOBS=8 --env PLATFORM=rhel9 --env TEST_RETRY= --env KSTESTS_KEEP=1 -v /var/tmp -v /home/rvykydal/work/git/rvykydal/kickstart-tests.devel/data:/opt/kstest/data:z -v /home/rvykydal/work/git/rvykydal/kickstart-tests.devel:/kickstart-tests:ro,z quay.io/rhinstaller/kstest-runner /bin/bash
[kstest@ebe52bb1a44b kstest]$ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 ; /kickstart-tests/scripts/probe_boot_iso.sh data/images/boot.iso
Parallel unsquashfs: Using 8 processors
1 inodes (24576 blocks) to write

created 1 file
created 2 directories
created 0 symlinks

... snip ...

libguestfs: launch libvirt guest
\x1bc\x1b[?7l\x1b[2J\x1b[0mSeaBIOS (version 1.16.2-1.fc38)
Machine UUID 7d82cea1-9cfc-4e38-97ad-549445c120bc
Booting from ROM..\x1bc\x1b[?7l\x1b[2J

hang

rvykydal commented 2 months ago

Seems we have too ancient container base image (F38), building container with F39 fixes the reproducer from previous comment to me. https://github.com/rhinstaller/kickstart-tests/pull/1132

rvykydal commented 2 months ago

Seems we have too ancient container base image (F38), building container with F39 fixes the reproducer from previous comment to me. #1132

The F39 container seems to have some serious issues for local use:

rvykydal commented 2 months ago

Closing with follow-up in the issues ^.