search

osbuild / image-builder

Image Builder service for console.redhat.com
Apache License 2.0
49 stars 44 forks source link

build edge image hang due to udevadm #206

Closed yih-redhat closed 3 years ago

yih-redhat commented 3 years ago

I was testing Edge related APIs against this PR: https://github.com/osbuild/image-builder/pull/193 And when I tried to build a commit image, it hangs at udevadm step.

Reproduce steps:

  1. Send a request to build Edge commit image and upload to s3.

    [root@vm02-iso image-builder]# curl -w %{http_code} -H 'x-rh-identity:  ${identity}' -H 'Content-Type: application/json' -d "@data.json" http://localhost:8086/api/image-builder/v1.0/compose
{"id":"a6193303-a3ff-442c-bb61-dd7f618d6d7a"}
201
[root@vm02-iso image-builder]# curl -w %{http_code} -H 'x-rh-identity:  ${identity}' http://localhost:8086/api/image-builder/v1.0/composes/a6193303-a3ff-442c-bb61-dd7f618d6d7a
{"image_status":{"status":"building"}}
200

  2. After hours the status is still building, then check the osbuild log:

    [root@vm02-iso image-builder]# pgrep osbuild-worker
919938
[root@vm02-iso image-builder]# pstree -al 919938
osbuild-worker -unix /run/osbuild-composer/job.socket
├─osbuild /usr/bin/osbuild --store /var/cache/osbuild-worker/osbuild-store --output-directory /var/cache/osbuild-worker/output/555fcc00-e663-4f96-a5a8-fa1fda042c56-877396296 --json - --export assembler
│   ├─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
│   │   └─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
│   │       └─org.osbuild.rhe /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
│   │           └─org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm
│   │               └─rpm --verbose --root /run/osbuild/tree --define _pkgverify_level none --install /tmp/manifest.06sulvm9
│   │                   └─sh /var/tmp/rpm-tmp.17hjYk 1
│   │                       └─udevadm trigger --sysname-match=hw_random --settle
│   └─4*[{osbuild}]
└─6*[{osbuild-worker}]

  3. kill the udevadm process and the building continues and will succeed soon.

    [root@vm02-iso image-builder]# pkill udevadm
[root@vm02-iso image-builder]# pstree -al 919938
osbuild-worker -unix /run/osbuild-composer/job.socket
├─osbuild /usr/bin/osbuild --store /var/cache/osbuild-worker/osbuild-store --output-directory /var/cache/osbuild-worker/output/555fcc00-e663-4f96-a5a8-fa1fda042c56-877396296 --json - --export assembler
│   ├─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
│   │   └─bwrap --cap-add CAP_MAC_ADMIN --chdir / --die-with-parent --new-session --setenv PATH /usr/sbin:/usr/bin --setenv PYTHONPATH /run/osbuild/lib --unshare-ipc --unshare-pid --unshare-net --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/boot /boot --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/usr /usr --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin --symlink usr/sbin /sbin --dev-bind /run/osbuild/osbuild-dev-ceufyz8e /dev --tmpfs /dev/shm --dir /etc --tmpfs /run --tmpfs /tmp --bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-var-dsejipmf /var --proc /proc --bind /sys /sys --ro-bind-try /sys/fs/selinux /sys/fs/selinux --ro-bind-try /var/cache/osbuild-worker/osbuild-store/tmp/tmpvk1wdi0mobject/tmpss4l4mhz-reader/etc/mke2fs.conf /etc/mke2fs.conf --ro-bind /usr/lib/osbuild /run/osbuild/lib --ro-bind /usr/lib/python3.6/site-packages/osbuild /run/osbuild/lib/osbuild --bind /var/cache/osbuild-worker/osbuild-store/tmp/tmpyyd8r56qobject/tmpzyrkrg1d-writer /run/osbuild/tree --ro-bind /usr/lib/osbuild/stages/org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/osbuild-sources-output-j5jxtcgj /run/osbuild/sources --ro-bind /var/cache/osbuild-worker/osbuild-store/tmp/store-server-934dvy1b/files-input-2ps1u69x /run/osbuild/inputs/packages --dir /run/osbuild/api --bind /run/osbuild/api-xi_o1jv3/osbuild /run/osbuild/api/osbuild --bind /run/osbuild/api-7ojiofx_/sources /run/osbuild/api/sources --bind /run/osbuild/api-vegeu_5k/remoteloop /run/osbuild/api/remoteloop -- /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
│   │       └─org.osbuild.rhe /run/osbuild/lib/runners/org.osbuild.rhel82 /run/osbuild/bin/org.osbuild.rpm
│   │           └─org.osbuild.rpm /run/osbuild/bin/org.osbuild.rpm
│   │               └─rpm --verbose --root /run/osbuild/tree --define _pkgverify_level none --install /tmp/manifest.06sulvm9
│   │                   └─sh /var/tmp/rpm-tmp.OGCbPk 0 0
│   │                       └─update-mime-dat -n /usr/share/mime
│   └─4*[{osbuild}]
└─6*[{osbuild-worker}]

I can always reproduce this bug, and @achilleas-k is aware of this issue, opening this issue to track it.

yih-redhat commented 3 years ago

Today I set up a new test environment with latest main branch of osbuild-composer and image-builder. This time I didn't see this issue when build commit tar image, but saw it again when build install ISO image.

Devel suspect it's caused by a special version of a package, hence I collected the package version here for future debug:

[root@rhel84-iso1 ~]# rpm -qa
iproute-tc-5.9.0-4.el8.x86_64
libXxf86misc-1.0.4-1.el8.x86_64
perl-DBD-SQLite-1.58-2.module+el8.1.0+2940+f62455ee.x86_64
systemd-udev-239-45.el8.x86_64
nautilus-sendto-3.8.6-2.el8.x86_64
liberation-fonts-common-2.00.3-7.el8.noarch
qemu-kvm-block-gluster-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
mesa-dri-drivers-20.3.3-2.el8.x86_64
e2fsprogs-1.45.6-1.el8.x86_64
polkit-libs-0.115-11.el8.x86_64
cyrus-sasl-plain-2.1.27-5.el8.x86_64
yelp-xsl-3.28.0-2.el8.noarch
flatpak-selinux-1.8.5-3.el8.noarch
gdisk-1.0.3-6.el8.x86_64
xdg-desktop-portal-1.6.0-4.el8.x86_64
gsettings-desktop-schemas-3.32.0-5.el8.x86_64
rsync-3.1.3-12.el8.x86_64
python3-pip-wheel-9.0.3-19.el8.noarch
python3-pydbus-0.6.0-5.el8.noarch
boost-atomic-1.66.0-10.el8.x86_64
libgdata-0.17.9-2.el8.x86_64
libnotify-0.7.7-5.el8.x86_64
google-noto-sans-lisu-fonts-20161022-7.el8.1.noarch
dejavu-sans-mono-fonts-2.35-7.el8.noarch
pcaudiolib-1.1-2.el8.x86_64
libdaemon-0.14-15.el8.x86_64
libcacard-2.7.0-2.el8_1.x86_64
parted-3.2-38.el8.x86_64
xorg-x11-xinit-session-1.3.4-18.el8.x86_64
vte-profile-0.52.4-2.el8.x86_64
python3-bytesize-1.4-3.el8.x86_64
sg3_utils-libs-1.44-5.el8.x86_64
poppler-utils-20.11.0-2.el8.x86_64
glusterfs-libs-6.0-49.1.el8.x86_64
man-pages-4.15-6.el8.x86_64
seabios-bin-1.13.0-2.module+el8.3.0+7353+9de0a3cc.noarch
iio-sensor-proxy-2.4-3.el8.x86_64
speex-1.2.0-1.el8.x86_64
gvfs-1.36.2-11.el8.x86_64
iproute-5.9.0-4.el8.x86_64
gnu-free-serif-fonts-20120503-18.el8.noarch
langtable-0.0.51-4.el8.noarch
libpeas-loader-python3-1.22.0-6.el8.x86_64
xorg-x11-xauth-1.0.9-12.el8.x86_64
libcanberra-0.30-18.el8.x86_64
bluez-5.52-4.el8.x86_64
lohit-gurmukhi-fonts-2.91.2-3.el8.noarch
adwaita-cursor-theme-3.28.0-2.el8.noarch
gstreamer1-plugins-base-1.16.1-2.el8.x86_64
libmpc-1.1.0-9.1.el8.x86_64
colord-1.4.2-1.el8.x86_64
plymouth-core-libs-0.9.4-9.20200615git1e36e30.el8.x86_64
sil-nuosu-fonts-2.1.1-14.el8.noarch
basesystem-11-5.el8.noarch
clutter-gtk-1.8.4-3.el8.x86_64
libXres-1.2.0-4.el8.x86_64
libvirt-daemon-driver-storage-iscsi-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
selinux-policy-3.14.3-67.el8.noarch
iwl7260-firmware-25.30.13.0-102.el8.1.noarch
ncurses-libs-6.1-7.20180224.el8.x86_64
libpeas-gtk-1.22.0-6.el8.x86_64
nss-softokn-freebl-3.53.1-17.el8_3.x86_64
python3-chardet-3.0.4-7.el8.noarch
geoclue2-libs-2.5.5-2.el8.x86_64
iwl2030-firmware-18.168.6.1-102.el8.1.noarch
libcom_err-1.45.6-1.el8.x86_64
gtkmm30-3.22.2-3.el8.x86_64
libcomps-0.1.11-5.el8.x86_64
dnf-plugin-subscription-manager-1.28.13-2.el8.x86_64
qemu-img-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
epel-release-8-10.el8.noarch
libgcrypt-1.8.5-4.el8.x86_64
gnome-session-3.28.1-10.el8.x86_64
libss-1.45.6-1.el8.x86_64
python3-bind-9.11.26-3.el8.noarch
openssh-8.0p1-5.el8.x86_64
httpd-tools-2.4.37-39.module+el8.4.0+9658+b87b2deb.x86_64
libpng-1.6.34-5.el8.x86_64
gnome-keyring-pam-3.28.2-1.el8.x86_64
pigz-2.4-4.el8.x86_64
containers-common-1.2.2-8.module+el8.4.0+10607+f4da7515.x86_64
plymouth-scripts-0.9.4-9.20200615git1e36e30.el8.x86_64
libtalloc-2.3.1-2.el8.x86_64
ibus-setup-1.5.19-12.el8.noarch
dosfstools-4.1-6.el8.x86_64
gnome-shell-extension-apps-menu-3.32.1-14.el8.noarch
gssdp-1.0.5-1.el8.x86_64
perl-Git-2.27.0-1.el8.noarch
json-c-0.13.1-0.4.el8.x86_64
plymouth-plugin-two-step-0.9.4-9.20200615git1e36e30.el8.x86_64
libnghttp2-1.33.0-3.el8_2.1.x86_64
python3-requests-ftp-0.3.1-11.el8.noarch
libblockdev-2.24-5.el8.x86_64
python3-cffi-1.11.5-5.el8.x86_64
nss-util-3.53.1-17.el8_3.x86_64
urw-base35-nimbus-roman-fonts-20170801-10.el8.noarch
libsigsegv-2.11-5.el8.x86_64
blivet-data-3.2.2-10.el8.noarch
pipewire-libs-0.3.6-1.el8.x86_64
libwayland-cursor-1.17.0-1.el8.x86_64
libspectre-0.2.8-5.el8.x86_64
lmdb-libs-0.9.24-1.el8.x86_64
python3-pexpect-4.3.1-3.el8.noarch
gpgme-1.13.1-7.el8.x86_64
libvirt-bash-completion-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
libdhash-0.5.0-39.el8.x86_64
libvirt-glib-3.0.0-1.el8.x86_64
fribidi-1.0.4-8.el8.x86_64
anaconda-gui-33.16.4.15-1.el8.x86_64
bolt-0.9.1-1.el8.x86_64
libXfixes-5.0.3-7.el8.x86_64
hplip-libs-3.18.4-9.el8.x86_64
libdatrie-0.2.9-7.el8.x86_64
cockpit-packagekit-238.2-1.el8.noarch
iputils-20180629-7.el8.x86_64
libogg-1.3.2-10.el8.x86_64
flatpak-session-helper-1.8.5-3.el8.x86_64
libhangul-0.1.0-16.el8.x86_64
tuned-2.15.0-2.el8.noarch
gnome-menus-3.13.3-11.el8.x86_64
fuse-libs-2.9.7-12.el8.x86_64
libvncserver-0.9.11-17.el8.x86_64
criu-3.15-1.module+el8.4.0+10607+f4da7515.x86_64
kpatch-0.9.2-3.el8.noarch
net-snmp-libs-5.8-20.el8.x86_64
podman-docker-2.2.1-7.module+el8.3.1+9857+68fb1526.noarch
libpciaccess-0.14-1.el8.x86_64
gupnp-av-0.12.10-6.el8.x86_64
pakchois-0.4-17.el8.x86_64
totem-nautilus-3.26.2-1.el8.x86_64
device-mapper-multipath-0.8.4-10.el8.x86_64
libsemanage-2.9-6.el8.x86_64
sssd-client-2.4.0-9.el8.x86_64
firewalld-filesystem-0.8.2-6.el8.noarch
gvfs-mtp-1.36.2-11.el8.x86_64
python3-audit-3.0-0.17.20191104git1c2f876.el8.x86_64
grub2-common-2.02-99.el8.noarch
cryptsetup-2.3.3-4.el8.x86_64
grub2-tools-minimal-2.02-99.el8.x86_64
enscript-1.6.6-17.el8.x86_64
libiscsi-1.18.0-8.module+el8.1.0+4066+0f1aadab.x86_64
libXv-1.0.11-7.el8.x86_64
iscsi-initiator-utils-iscsiuio-6.2.1.2-1.gita8fcb37.el8.x86_64
grubby-8.40-41.el8.x86_64
libsane-hpaio-3.18.4-9.el8.x86_64
jose-10-2.el8.x86_64
dmidecode-3.2-8.el8.x86_64
libvirt-daemon-driver-network-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
crypto-policies-scripts-20210209-1.gitbfb6bed.el8_3.noarch
eog-3.28.4-1.el8.x86_64
ndctl-71.1-2.el8.x86_64
protobuf-c-1.3.0-6.el8.x86_64
perl-Digest-MD5-2.55-396.el8.x86_64
libtirpc-1.1.4-4.el8.x86_64
baobab-3.28.0-4.el8.x86_64
pulseaudio-module-bluetooth-14.0-2.el8.x86_64
osbuild-28-1.20210408giteb74ddf.20210408giteb74ddf.el8.noarch
exiv2-libs-0.27.3-2.el8.x86_64
perl-Mozilla-CA-20160104-7.module+el8.3.0+6498+9eecfe51.noarch
gettext-libs-0.19.8.1-17.el8.x86_64
firewalld-0.8.2-6.el8.noarch
usermode-1.113-1.el8.x86_64
fstrm-0.6.0-3.el8.1.x86_64
perl-Pod-Perldoc-3.28-396.el8.noarch
shadow-utils-4.6-12.el8.x86_64
xorg-x11-drv-intel-2.99.917-39.20200205.el8.x86_64
xorg-x11-server-Xorg-1.20.10-1.el8.x86_64
libglvnd-egl-1.3.2-1.el8.x86_64
perl-Encode-2.97-3.el8.x86_64
kmod-25-17.el8.x86_64
rsyslog-gssapi-8.1911.0-7.el8.x86_64
gssproxy-0.8.0-19.el8.x86_64
xorg-x11-server-common-1.20.10-1.el8.x86_64
perl-Unicode-Normalize-1.25-396.el8.x86_64
util-linux-2.32.1-27.el8.x86_64
spice-vdagent-0.20.0-3.el8.x86_64
libmbim-utils-1.20.2-1.el8.x86_64
groff-base-1.22.3-18.el8.x86_64
glib2-2.56.4-9.el8.x86_64
google-noto-fonts-common-20161022-7.el8.1.noarch
smartmontools-7.1-1.el8.x86_64
glusterfs-6.0-49.1.el8.x86_64
cockpit-ws-238.2-1.el8.x86_64
xmlsec1-1.2.25-4.el8.x86_64
atk-2.28.1-1.el8.x86_64
hyperv-daemons-license-0-0.30.20180415git.el8.noarch
prefixdevname-0.1.0-6.el8.x86_64
python3-policycoreutils-2.9-14.el8.noarch
chrony-3.5-2.el8.x86_64
boost-iostreams-1.66.0-10.el8.x86_64
gstreamer1-1.16.1-2.el8.x86_64
geolite2-country-20180605-1.el8.noarch
sg3_utils-1.44-5.el8.x86_64
libblockdev-kbd-2.24-5.el8.x86_64
xorg-x11-drv-wacom-serial-support-0.38.0-1.el8.x86_64
e2fsprogs-libs-1.45.6-1.el8.x86_64
cups-libs-2.2.6-38.el8.x86_64
adobe-mappings-cmap-deprecated-20171205-3.el8.noarch
lshw-B.02.19.2-5.el8.x86_64
python3-cups-1.9.72-21.el8.x86_64
poppler-glib-20.11.0-2.el8.x86_64
libfontenc-1.1.3-8.el8.x86_64
gvfs-client-1.36.2-11.el8.x86_64
google-noto-emoji-color-fonts-20180508-4.el8.noarch
google-noto-sans-tai-tham-fonts-20161022-7.el8.1.noarch
pulseaudio-utils-14.0-2.el8.x86_64
qemu-kvm-block-rbd-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
libini_config-1.3.1-39.el8.x86_64
avahi-glib-0.7-20.el8.x86_64
tracer-common-0.7.5-2.el8.noarch
symlinks-1.4-19.el8.x86_64
libsamplerate-0.1.9-1.el8.x86_64
libfprint-1.90.7-1.el8.x86_64
libevdev-1.8.0-1.el8.x86_64
libpeas-1.22.0-6.el8.x86_64
man-pages-overrides-8.3.0.2-2.el8.noarch
google-noto-serif-cjk-ttc-fonts-20190416-1.el8.noarch
gnome-video-effects-0.4.3-3.el8.noarch
totem-3.26.2-1.el8.x86_64
zip-3.0-23.el8.x86_64
libwacom-1.6-2.el8.x86_64
gnome-user-docs-3.28.2-1.el8.noarch
lohit-bengali-fonts-2.91.5-3.el8.noarch
libXft-2.3.3-1.el8.x86_64
gnome-bluetooth-libs-3.34.3-1.el8.x86_64
ipcalc-0.2.4-4.el8.x86_64
libmodulemd-2.9.4-2.el8.x86_64
redhat-release-eula-8.4-0.6.el8.x86_64
paktype-naskh-basic-fonts-4.1-9.el8.noarch
librsvg2-2.42.7-4.el8.x86_64
quota-4.04-12.el8.x86_64
vim-minimal-8.0.1763-15.el8.x86_64
xmlrpc-c-client-1.51.0-5.el8.x86_64
publicsuffix-list-dafsa-20180723-1.el8.noarch
rootfiles-8.1-22.el8.noarch
nautilus-extensions-3.28.1-15.el8.x86_64
libvirt-daemon-driver-storage-mpath-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
ipset-libs-7.1-1.el8.x86_64
accountsservice-libs-0.6.55-1.el8.x86_64
glibc-2.28-151.el8.x86_64
iwl5150-firmware-8.24.2.2-102.el8.1.noarch
colord-gtk-0.1.26-8.el8.x86_64
python3-inotify-0.9.6-13.el8.noarch
lua-json-1.3.2-9.el8.noarch
rest-0.8.1-2.el8.x86_64
libcap-ng-0.7.9-5.el8.x86_64
iwl100-firmware-39.31.5.1-102.el8.1.noarch
adwaita-gtk2-theme-3.22.3-4.el8.x86_64
python3-meh-0.47.2-1.el8.noarch
libidn-1.34-5.el8.x86_64
libgtop2-2.38.0-3.el8.x86_64
popt-1.18-1.el8.x86_64
python3-jmespath-0.9.0-11.el8.noarch
usermode-gtk-1.113-1.el8.x86_64
sssd-ipa-2.4.0-9.el8.x86_64
libvpx-1.7.0-8.el8.x86_64
samba-common-4.13.3-3.el8.noarch
numactl-libs-2.0.12-11.el8.x86_64
python3-markupsafe-0.23-19.el8.x86_64
anaconda-widgets-33.16.4.15-1.el8.x86_64
system-config-printer-libs-1.5.11-13.el8.noarch
xdg-user-dirs-0.17-1.el8.x86_64
hicolor-icon-theme-0.17-2.el8.noarch
nspr-4.25.0-2.el8_2.x86_64
python3-jinja2-2.10.1-2.el8_0.noarch
vino-3.22.0-10.el8.x86_64
gnome-shell-extension-launch-new-instance-3.32.1-14.el8.noarch
hdparm-9.54-3.el8.x86_64
libreport-web-2.9.5-15.el8.x86_64
p11-kit-0.23.22-1.el8.x86_64
mod_http2-1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
urw-base35-d050000l-fonts-20170801-10.el8.noarch
sssd-2.4.0-9.el8.x86_64
xorg-x11-font-utils-7.5-40.el8.x86_64
libblockdev-loop-2.24-5.el8.x86_64
libzstd-1.4.4-1.el8.x86_64
python3-pynacl-1.3.0-5.el8.x86_64
urw-base35-fonts-20170801-10.el8.noarch
python3-louis-2.6.2-21.el8.noarch
libsss_nss_idmap-2.4.0-9.el8.x86_64
python3-iniparse-0.4-31.el8.noarch
libcollection-0.7.0-39.el8.x86_64
python3-libvirt-6.0.0-1.module+el8.3.0+6423+e4cb6418.x86_64
authselect-libs-1.2.2-2.el8.x86_64
subscription-manager-cockpit-1.28.13-2.el8.noarch
slang-2.3.2-3.el8.x86_64
PackageKit-1.1.12-6.el8.x86_64
libmnl-1.0.4-6.el8.x86_64
virt-manager-2.2.1-4.el8.noarch
sane-backends-libs-1.0.27-22.el8.x86_64
initial-setup-gui-0.3.81.7-1.el8.x86_64
jbigkit-libs-2.1-14.el8.x86_64
gtk-update-icon-cache-3.22.30-6.el8.x86_64
libXcomposite-0.4.4-14.el8.x86_64
wpa_supplicant-2.9-5.el8.x86_64
qemu-kvm-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
libfastjson-0.99.8-2.el8.x86_64
python3-libstoragemgmt-1.8.7-1.el8.noarch
libXcursor-1.1.15-3.el8.x86_64
oddjob-mkhomedir-0.34.7-1.el8.x86_64
gnome-initial-setup-3.28.0-9.el8.x86_64
xorg-x11-server-utils-7.7-27.el8.x86_64
libvirt-gconfig-3.0.0-1.el8.x86_64
libacl-2.2.53-1.el8.x86_64
libvarlink-18-3.el8.x86_64
enchant-1.6.0-21.el8.x86_64
cheese-3.28.0-3.el8.x86_64
ncompress-4.2.4.4-13.el8.x86_64
bind-libs-lite-9.11.26-3.el8.x86_64
libsss_idmap-2.4.0-9.el8.x86_64
slirp4netns-1.1.8-1.module+el8.4.0+10607+f4da7515.x86_64
gvfs-afc-1.36.2-11.el8.x86_64
libssh-config-0.9.4-2.el8.noarch
binutils-2.30-93.el8.x86_64
hunspell-en-GB-0.20140811.1-12.el8.noarch
sssd-ldap-2.4.0-9.el8.x86_64
sushi-3.28.3-1.el8.x86_64
platform-python-pip-9.0.3-19.el8.noarch
python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64
lzo-2.08-14.el8.x86_64
rpm-plugin-systemd-inhibit-4.14.3-13.el8.x86_64
grub2-pc-2.02-99.el8.x86_64
libssh-0.9.4-2.el8.x86_64
sssd-nfs-idmap-2.4.0-9.el8.x86_64
xorg-x11-xkb-utils-7.7-28.el8.x86_64
libmusicbrainz5-5.1.0-10.el8.x86_64
ibus-hangul-1.5.1-6.el8.x86_64
curl-7.61.1-18.el8.x86_64
python3-rpm-4.14.3-13.el8.x86_64
pciutils-libs-3.7.0-1.el8.x86_64
libvirt-daemon-driver-qemu-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
gnome-system-monitor-3.28.2-1.el8.x86_64
cracklib-dicts-2.9.6-15.el8.x86_64
audit-3.0-0.17.20191104git1c2f876.el8.x86_64
libtheora-1.1.1-21.el8.x86_64
python3-jsonschema-2.6.0-4.el8.noarch
perl-Pod-Escapes-1.07-395.el8.noarch
gtk3-immodule-xim-3.22.30-6.el8.x86_64
elfutils-libs-0.182-3.el8.x86_64
dleyna-connector-dbus-0.3.0-2.el8.x86_64
libcdio-2.0.0-3.el8.x86_64
perl-Pod-Simple-3.35-395.el8.noarch
vdo-6.2.4.14-14.el8.x86_64
dbus-tools-1.12.8-12.el8.x86_64
libosinfo-1.9.0-1.el8.x86_64
libglvnd-1.3.2-1.el8.x86_64
perl-Getopt-Long-2.50-4.el8.noarch
xorg-x11-drv-vmware-13.2.1-8.el8.x86_64
kmod-libs-25-17.el8.x86_64
device-mapper-event-1.02.175-5.el8.x86_64
openjpeg2-2.3.1-6.el8.x86_64
perl-parent-0.237-1.el8.noarch
iptstate-2.2.6-6.el8.x86_64
kbd-2.0.4-10.el8.x86_64
cups-pk-helper-0.2.6-5.el8.x86_64
kyotocabinet-libs-1.2.76-17.el8.x86_64
perl-threads-2.21-2.el8.x86_64
mcelog-173-0.el8.x86_64
grub2-tools-2.02-99.el8.x86_64
acl-2.2.53-1.el8.x86_64
fontpackages-filesystem-1.44-22.el8.noarch
harfbuzz-icu-1.7.5-3.el8.x86_64
dbus-1.12.8-12.el8.x86_64
startup-notification-0.12-15.el8.x86_64
tzdata-2021a-1.el8.noarch
gdk-pixbuf2-2.36.12-5.el8.x86_64
boost-regex-1.66.0-10.el8.x86_64
gnu-free-fonts-common-20120503-18.el8.noarch
cairo-gobject-1.15.12-3.el8.x86_64
libmetalink-0.1.3-7.el8.x86_64
bind-license-9.11.26-3.el8.noarch
libgudev-232-4.el8.x86_64
diffutils-3.6-6.el8.x86_64
google-noto-cjk-fonts-common-20190416-1.el8.noarch
python3-six-1.11.0-8.el8.noarch
xml-common-0.6.3-50.el8.noarch
rhsm-icons-1.28.13-2.el8.noarch
avahi-libs-0.7-20.el8.x86_64
mozjs60-60.9.0-4.el8.x86_64
emacs-filesystem-26.1-5.el8.noarch
python3-gobject-base-3.28.3-2.el8.x86_64
boost-program-options-1.66.0-10.el8.x86_64
liberation-sans-fonts-2.00.3-7.el8.noarch
NetworkManager-1.30.0-7.el8.x86_64
libXfont2-2.0.3-2.el8.x86_64
google-droid-sans-fonts-20120715-13.el8.noarch
satyr-0.26-2.el8.x86_64
libnfnetlink-1.0.1-13.el8.x86_64
smc-fonts-common-6.1-10.el8.noarch
python3-libselinux-2.9-5.el8.x86_64
libyaml-0.1.7-5.el8.x86_64
unicode-ucd-11.0.0-1.el8.noarch
webkit2gtk3-jsc-2.30.4-1.el8.x86_64
dotconf-1.3-18.el8.x86_64
sgabios-bin-0.20170427git-3.module+el8.1.0+4066+0f1aadab.noarch
iptables-1.8.4-17.el8.x86_64
libpmem-1.6.1-1.el8.x86_64
mesa-filesystem-20.3.3-2.el8.x86_64
harfbuzz-1.7.5-3.el8.x86_64
gutenprint-5.2.14-3.el8.x86_64
libkkc-common-0.3.5-12.el8.noarch
libmbim-1.20.2-1.el8.x86_64
lua-socket-3.0-0.17.rc1.el8.x86_64
hplip-common-3.18.4-9.el8.x86_64
cairomm-1.12.0-8.el8.x86_64
libavc1394-0.5.4-7.el8.x86_64
appstream-data-8-20200724.el8.noarch
python3-dbus-1.2.4-15.el8.x86_64
libdvdnav-5.0.3-8.el8.x86_64
subscription-manager-rhsm-certificates-1.28.13-2.el8.x86_64
xmlrpc-c-1.51.0-5.el8.x86_64
squashfs-tools-4.3-20.el8.x86_64
filesystem-3.8-3.el8.x86_64
python3-libxml2-2.9.7-9.el8.x86_64
attr-2.4.48-3.el8.x86_64
quota-nls-4.04-12.el8.noarch
atkmm-2.24.2-7.el8.x86_64
libdmx-1.1.4-3.el8.x86_64
pcre2-10.32-2.el8.x86_64
accountsservice-0.6.55-1.el8.x86_64
ipset-7.1-1.el8.x86_64
glibc-common-2.28-151.el8.x86_64
cockpit-bridge-238.2-1.el8.x86_64
keyutils-1.5.10-6.el8.x86_64
zlib-1.2.11-17.el8.x86_64
geocode-glib-3.26.0-3.el8.x86_64
bc-1.07.1-5.el8.x86_64
libuuid-2.32.1-27.el8.x86_64
libimobiledevice-1.2.0-16.el8.x86_64
coreutils-common-8.30-8.el8.x86_64
xz-libs-5.2.4-3.el8.x86_64
enchant2-2.2.3-3.el8.x86_64
liblouis-2.6.2-21.el8.x86_64
libnl3-3.5.0-1.el8.x86_64
libudisks2-2.9.0-6.el8.x86_64
libmodman-2.0.1-17.el8.x86_64
bzip2-libs-1.0.6-26.el8.x86_64
logrotate-3.14.0-4.el8.x86_64
marisa-0.2.4-36.el8.x86_64
expat-2.2.5-4.el8.x86_64
samba-client-libs-4.13.3-3.el8.x86_64
less-530-1.el8.x86_64
chkconfig-1.13-2.el8.x86_64
rsyslog-8.1911.0-7.el8.x86_64
c-ares-1.13.0-5.el8.x86_64
libICE-1.0.9-15.el8.x86_64
python3-pyyaml-3.12-12.el8.x86_64
fuse3-3.2.1-12.el8.x86_64
lua-libs-5.3.4-11.el8.x86_64
dleyna-core-0.6.0-3.el8.x86_64
jimtcl-0.77-6.el8.x86_64
libffi-3.1-22.el8.x86_64
GConf2-3.2.6-22.el8.x86_64
libpkgconf-1.4.2-1.el8.x86_64
libseccomp-2.5.1-1.el8.x86_64
libblockdev-fs-2.24-5.el8.x86_64
libquvi-scripts-0.9.20131130-9.el8.noarch
libtool-ltdl-2.4.6-25.el8.x86_64
python3-pyparted-3.11.0-13.el8.x86_64
m17n-db-1.8.0-3.el8.noarch
libwayland-egl-1.17.0-1.el8.x86_64
pulseaudio-14.0-2.el8.x86_64
libsss_sudo-2.4.0-9.el8.x86_64
libbasicobjects-0.1.1-39.el8.x86_64
gnupg2-smime-2.2.20-2.el8.x86_64
lzo-minilzo-2.08-14.el8.x86_64
gmp-6.1.2-10.el8.x86_64
libdnf-0.55.0-7.el8.x86_64
newt-0.52.20-11.el8.x86_64
augeas-libs-1.12.0-6.el8.x86_64
python3-hawkey-0.55.0-7.el8.x86_64
gpm-libs-1.20.7-17.el8.x86_64
libX11-1.6.8-4.el8.x86_64
gdk-pixbuf2-modules-2.36.12-5.el8.x86_64
libtiff-4.0.9-18.el8.x86_64
libXdamage-1.1.4-14.el8.x86_64
cronie-anacron-1.5.2-4.el8.x86_64
m17n-lib-1.8.0-2.el8.x86_64
libXtst-1.2.3-7.el8.x86_64
python3-libstoragemgmt-clibs-1.8.7-1.el8.x86_64
libgdither-0.6-17.el8.x86_64
libvorbis-1.3.6-2.el8.x86_64
at-spi2-atk-2.26.2-1.el8.x86_64
libiptcdata-1.0.4-21.el8.x86_64
file-libs-5.33-16.el8_3.1.x86_64
libgexiv2-0.10.8-4.el8.x86_64
libmspack-0.7-0.3.alpha.el8.4.x86_64
libattr-2.4.48-3.el8.x86_64
daxctl-libs-71.1-2.el8.x86_64
libusal-1.1.11-39.el8.x86_64
libunistring-0.9.9-3.el8.x86_64
ima-evm-utils-1.3.2-12.el8.x86_64
oniguruma-6.8.2-2.el8.x86_64
mesa-libgbm-20.3.3-2.el8.x86_64
device-mapper-multipath-libs-0.8.4-10.el8.x86_64
wavpack-5.1.0-15.el8.x86_64
libepoxy-1.5.3-1.el8.x86_64
python3-systemd-234-8.el8.x86_64
kbd-misc-2.0.4-10.el8.noarch
boost-system-1.66.0-10.el8.x86_64
NetworkManager-team-1.30.0-7.el8.x86_64
dbus-common-1.12.8-12.el8.noarch
hunspell-1.6.2-1.el8.x86_64
python3-pip-9.0.3-19.el8.noarch
platform-python-setuptools-39.2.0-6.el8.noarch
p11-kit-trust-0.23.22-1.el8.x86_64
nftables-0.9.3-18.el8.x86_64
libibverbs-32.0-4.el8.x86_64
tar-1.30-5.el8.x86_64
python3-nftables-0.9.3-18.el8.x86_64
openldap-2.4.46-16.el8.x86_64
libxkbfile-1.1.0-1.el8.x86_64
qemu-kvm-block-ssh-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
libkcapi-hmaccalc-1.2.0-2.el8.x86_64
adwaita-icon-theme-3.28.0-2.el8.noarch
rpm-build-libs-4.14.3-13.el8.x86_64
libcurl-7.61.1-18.el8.x86_64
libsmartcols-2.32.1-27.el8.x86_64
tpm2-tools-4.1.1-2.el8.x86_64
elfutils-default-yama-scope-0.182-3.el8.noarch
libmaxminddb-1.2.0-10.el8.x86_64
libkkc-0.3.5-12.el8.x86_64
procps-ng-3.3.15-6.el8.x86_64
flac-libs-1.3.2-9.el8.x86_64
python3-librepo-1.12.0-3.el8.x86_64
kpartx-0.8.4-10.el8.x86_64
libical-3.0.3-3.el8.x86_64
libreport-anaconda-2.9.5-15.el8.x86_64
openssl-pkcs11-0.4.10-2.el8.x86_64
bzip2-1.0.6-26.el8.x86_64
libzhuyin-2.2.0-1.el8.x86_64
libfdisk-2.32.1-27.el8.x86_64
libverto-0.3.0-5.el8.x86_64
osinfo-db-tools-1.9.0-1.el8.x86_64
coreutils-8.30-8.el8.x86_64
libsndfile-1.0.28-10.el8.x86_64
xorg-x11-drv-libinput-0.29.0-1.el8.x86_64
device-mapper-libs-1.02.175-5.el8.x86_64
libwebp-1.0.0-1.el8.x86_64
itstool-2.0.6-2.el8.noarch
libdb-5.3.28-40.el8.x86_64
libglvnd-gles-1.3.2-1.el8.x86_64
libblockdev-lvm-2.24-5.el8.x86_64
dbus-daemon-1.12.8-12.el8.x86_64
libglvnd-glx-1.3.2-1.el8.x86_64
python3-unbound-1.7.3-15.el8.x86_64
libpwquality-1.4.4-3.el8.x86_64
lzop-1.03-20.el8.x86_64
kernel-modules-4.18.0-304.el8.x86_64
dracut-049-135.git20210121.el8.x86_64
boost-thread-1.66.0-10.el8.x86_64
mlocate-0.26-20.el8.x86_64
perl-Math-BigInt-1.9998.11-7.el8.noarch
qemu-guest-agent-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
sscg-2.3.3-14.el8.x86_64
glusterfs-client-xlators-6.0-49.1.el8.x86_64
nvme-cli-1.12-3.el8.x86_64
xorg-x11-server-Xwayland-1.20.10-1.el8.x86_64
rpcbind-1.2.5-8.el8.x86_64
man-db-2.7.6.1-17.el8.x86_64
dhcp-client-4.3.6-44.el8.x86_64
policycoreutils-python-utils-2.9-14.el8.noarch
bash-completion-2.7-5.el8.noarch
p11-kit-server-0.23.22-1.el8.x86_64
python3-slip-dbus-0.6.4-11.el8.noarch
biosdevname-0.7.3-2.el8.x86_64
libatasmart-0.19-14.el8.x86_64
libreport-cli-2.9.5-15.el8.x86_64
bpftool-4.18.0-304.el8.x86_64
liboauth-1.0.3-9.el8.x86_64
cups-client-2.2.6-38.el8.x86_64
iprutils-2.4.19-1.el8.x86_64
poppler-20.11.0-2.el8.x86_64
avahi-0.7-20.el8.x86_64
google-noto-sans-meetei-mayek-fonts-20161022-7.el8.1.noarch
evolution-data-server-langpacks-3.28.5-15.el8.noarch
speech-dispatcher-espeak-ng-0.8.8-6.el8.x86_64
google-noto-sans-tai-viet-fonts-20161022-7.el8.1.noarch
spice-gtk3-0.38-6.el8.x86_64
python3-configobj-5.0.6-11.el8.noarch
libsysfs-2.1.0-24.el8.x86_64
tracker-2.1.5-2.el8.x86_64
brlapi-0.6.7-32.el8.x86_64
tree-1.7.0-15.el8.x86_64
cups-2.2.6-38.el8.x86_64
alsa-utils-1.2.4-2.el8.x86_64
smc-meera-fonts-6.1-10.el8.noarch
libblockdev-crypto-2.24-5.el8.x86_64
usb_modeswitch-data-20191128-1.el8.noarch
gnu-free-mono-fonts-20120503-18.el8.noarch
gvfs-fuse-1.36.2-11.el8.x86_64
python3-cairo-1.16.3-6.el8.x86_64
jomolhari-fonts-0.003-24.el8.noarch
udisks2-lvm2-2.9.0-6.el8.x86_64
setroubleshoot-server-3.3.24-3.el8.x86_64
lohit-devanagari-fonts-2.95.4-3.el8.noarch
cheese-libs-3.28.0-3.el8.x86_64
pango-1.42.4-6.el8.x86_64
lohit-odia-fonts-2.91.2-3.el8.noarch
mutter-3.32.2-57.el8.x86_64
cogl-1.22.2-10.el8.x86_64
paratype-pt-sans-fonts-20141121-6.el8.noarch
lsof-4.93.2-1.el8.x86_64
webkit2gtk3-2.30.4-1.el8.x86_64
stix-fonts-1.1.0-12.el8.noarch
libvirt-daemon-driver-storage-disk-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
libnma-1.8.30-2.el8.x86_64
mailcap-2.1.48-3.el8.noarch
libvirt-daemon-driver-storage-logical-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
gtksourceview3-3.24.9-1.el8.x86_64
iwl6000g2b-firmware-18.168.6.1-102.el8.1.noarch
libvirt-daemon-driver-storage-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
zenity-3.28.1-1.el8.x86_64
iwl5000-firmware-8.83.5.1_1-102.el8.1.noarch
python3-idna-2.5-5.el8.noarch
gspell-1.8.1-1.el8.x86_64
iwl135-firmware-18.168.6.1-102.el8.1.noarch
dnf-4.4.2-11.el8.noarch
yelp-libs-3.28.1-3.el8.x86_64
alsa-sof-firmware-1.6.1-2.el8.noarch
python3-libreport-2.9.5-15.el8.x86_64
gnome-themes-standard-3.22.3-4.el8.x86_64
apr-util-bdb-1.6.1-6.el8.x86_64
python3-perf-4.18.0-304.el8.x86_64
ibus-gtk2-1.5.19-12.el8.x86_64
git-core-2.27.0-1.el8.x86_64
sssd-ad-2.4.0-9.el8.x86_64
rhsm-gtk-1.28.13-2.el8.x86_64
python3-requests-2.20.0-2.1.el8_1.noarch
gupnp-dlna-0.10.5-9.el8.x86_64
gnome-autoar-0.2.3-1.el8.x86_64
gnome-shell-3.32.2-30.el8.x86_64
keybinder3-0.3.2-4.el8.x86_64
perl-TermReadKey-2.37-7.el8.x86_64
gnome-shell-extension-horizontal-workspaces-3.32.1-14.el8.noarch
paps-libs-0.6.8-41.el8.x86_64
python3-kickstart-3.16.11-1.el8.noarch
plymouth-theme-charge-0.9.4-9.20200615git1e36e30.el8.x86_64
httpd-2.4.37-39.module+el8.4.0+9658+b87b2deb.x86_64
python3-sssdconfig-2.4.0-9.el8.noarch
urw-base35-gothic-fonts-20170801-10.el8.noarch
python3-bcrypt-3.1.6-2.el8.1.x86_64
authselect-compat-1.2.2-2.el8.x86_64
urw-base35-p052-fonts-20170801-10.el8.noarch
python3-paramiko-2.4.3-1.el8.noarch
python3-langtable-0.0.51-4.el8.noarch
libgs-9.27-1.el8.x86_64
skopeo-1.2.2-8.module+el8.4.0+10607+f4da7515.x86_64
python3-psutil-5.4.3-10.el8.x86_64
gtk-vnc2-0.9.0-2.el8.x86_64
python3-argcomplete-1.9.3-6.el8.noarch
cockpit-system-238.2-1.el8.noarch
cyrus-sasl-2.1.27-5.el8.x86_64
gnutls-utils-3.6.14-9.el8.x86_64
anaconda-tui-33.16.4.15-1.el8.x86_64
lockdev-1.0.4-0.28.20111007git.el8.x86_64
subscription-manager-initial-setup-addon-1.28.13-2.el8.x86_64
sane-backends-1.0.27-22.el8.x86_64
python3-lxml-4.2.3-2.el8.x86_64
libblockdev-mdraid-2.24-5.el8.x86_64
qemu-kvm-core-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
NetworkManager-wifi-1.30.0-7.el8.x86_64
cockpit-238.2-1.el8.x86_64
hypervkvpd-0-0.30.20180415git.el8.x86_64
chrome-gnome-shell-10.1-7.el8.x86_64
radvd-2.17-15.el8.x86_64
buildah-1.19.7-1.module+el8.4.0+10607+f4da7515.x86_64
libgcab1-1.1-1.el8.x86_64
podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64
yum-4.4.2-11.el8.noarch
python3-enchant-2.0.0-3.el8.noarch
gnome-screenshot-3.26.0-3.el8.x86_64
libquvi-0.9.4-12.el8.x86_64
gedit-3.28.1-3.el8.x86_64
grub2-tools-extra-2.02-99.el8.x86_64
gvfs-goa-1.36.2-11.el8.x86_64
sssd-krb5-common-2.4.0-9.el8.x86_64
gutenprint-cups-5.2.14-3.el8.x86_64
sssd-proxy-2.4.0-9.el8.x86_64
open-vm-tools-desktop-11.2.0-2.el8.x86_64
libluksmeta-9-4.el8.x86_64
sssd-kcm-2.4.0-9.el8.x86_64
bind-export-libs-9.11.26-3.el8.x86_64
rsyslog-relp-8.1911.0-7.el8.x86_64
libsrtp-1.5.4-8.el8.x86_64
paps-0.6.8-41.el8.x86_64
nmap-ncat-7.70-5.el8.x86_64
ibus-libzhuyin-1.8.93-1.el8.x86_64
libvirt-daemon-driver-interface-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
gnome-terminal-nautilus-3.28.3-3.el8.x86_64
libvirt-daemon-driver-secret-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
PackageKit-gtk3-module-1.1.12-6.el8.x86_64
perl-libnet-3.11-3.el8.noarch
python3-osbuild-28-1.20210408giteb74ddf.20210408giteb74ddf.el8.noarch
gnome-logs-3.28.5-3.el8.x86_64
perl-Time-Local-1.280-1.el8.noarch
osbuild-composer-worker-30-1.20210617git2ed4dde.el8.x86_64
alsa-plugins-pulseaudio-1.1.9-1.el8.x86_64
perl-Term-ANSIColor-4.06-396.el8.noarch
NetworkManager-bluetooth-1.30.0-7.el8.x86_64
perl-HTTP-Tiny-0.074-1.el8.noarch
xorg-x11-drv-ati-19.1.0-1.el8.x86_64
perl-Pod-Usage-1.69-395.el8.noarch
xorg-x11-drv-qxl-0.1.5-11.el8.x86_64
perl-Errno-1.28-419.el8.x86_64
wget-1.19.5-10.el8.x86_64
perl-Exporter-5.72-396.el8.noarch
util-linux-user-2.32.1-27.el8.x86_64
perl-macros-5.26.3-419.el8.x86_64
NetworkManager-tui-1.30.0-7.el8.x86_64
perl-IO-1.38-419.el8.x86_64
at-3.1.20-11.el8.x86_64
perl-threads-shared-1.58-2.el8.x86_64
NetworkManager-wwan-1.30.0-7.el8.x86_64
libgcc-8.4.1-1.el8.x86_64
device-mapper-persistent-data-0.8.5-4.el8.x86_64
xkeyboard-config-2.28-1.el8.noarch
ethtool-5.8-5.el8.x86_64
trousers-0.3.15-1.el8.x86_64
abattis-cantarell-fonts-0.0.25-6.el8.noarch
llvm-libs-11.0.0-2.module+el8.4.0+8598+a071fcd5.x86_64
cairo-1.15.12-3.el8.x86_64
cups-filesystem-2.2.6-38.el8.noarch
exempi-2.4.5-2.el8.x86_64
libusbx-1.0.23-4.el8.x86_64
mozilla-filesystem-1.9-18.el8.x86_64
libgomp-8.4.1-1.el8.x86_64
alsa-lib-1.2.4-5.el8.x86_64
adobe-mappings-cmap-20171205-3.el8.noarch
iso-codes-3.79-2.el8.noarch
pulseaudio-libs-14.0-2.el8.x86_64
fuse-common-3.2.1-12.el8.x86_64
snappy-1.1.8-3.el8.x86_64
polkit-pkla-compat-0.1-12.el8.x86_64
liberation-mono-fonts-2.00.3-7.el8.noarch
taglib-1.11.1-8.el8.x86_64
NetworkManager-libnm-1.30.0-7.el8.x86_64
osinfo-db-20210215-1.el8.noarch
brotli-1.0.6-3.el8.x86_64
dconf-0.28.0-4.el8.x86_64
khmeros-fonts-common-5.0-25.el8.noarch
libnetfilter_conntrack-1.0.6-5.el8.x86_64
libblockdev-utils-2.24-5.el8.x86_64
vim-filesystem-8.0.1763-15.el8.noarch
ncurses-6.1-7.20180224.el8.x86_64
ModemManager-glib-1.10.8-2.el8.x86_64
skkdic-20170102-4.T1100.el8.noarch
gutenprint-libs-5.2.14-3.el8.x86_64
redhat-logos-84.4-1.el8.x86_64
poppler-data-0.4.9-1.el8.noarch
mtdev-1.1.5-12.el8.x86_64
glibmm24-2.56.0-2.el8.x86_64
libkkc-data-0.2.7-12.el8.x86_64
woff2-1.0.2-5.el8.x86_64
pulseaudio-libs-glib2-14.0-2.el8.x86_64
ipxe-roms-qemu-20181214-8.git133f4c47.el8.noarch
mesa-libxatracker-20.3.3-2.el8.x86_64
colord-libs-1.4.2-1.el8.x86_64
cldr-emoji-annotation-33.1.0_0-1.el8.noarch
libiec61883-1.2.0-18.el8.x86_64
dbus-glib-0.110-2.el8.x86_64
adobe-mappings-pdf-20180407-1.el8.noarch
gdbm-1.18-1.el8.x86_64
libsss_certmap-2.4.0-9.el8.x86_64
setup-2.12.2-6.el8.noarch
grub2-pc-modules-2.02-99.el8.noarch
python3-decorator-4.2.1-2.el8.noarch
redhat-indexhtml-8-7.el8.noarch
celt051-0.5.1.3-15.el8.x86_64
libqmi-1.24.0-1.el8.x86_64
ncurses-base-6.1-7.20180224.el8.noarch
libXpm-3.5.12-8.el8.x86_64
selinux-policy-targeted-3.14.3-67.el8.noarch
glibc-langpack-en-2.28-151.el8.x86_64
netcf-libs-0.2.8-12.module+el8.1.0+4066+0f1aadab.x86_64
libsoup-2.62.3-2.el8.x86_64
libsepol-2.9-2.el8.x86_64
lua-expat-1.3.0-12.el8.1.x86_64
libappstream-glib-0.7.14-3.el8.x86_64
libgpg-error-1.31-1.el8.x86_64
sqlite-3.26.0-13.el8.x86_64
librhsm-0.0.3-4.el8.x86_64
libcap-2.26-4.el8.x86_64
mtools-4.0.18-14.el8.x86_64
libuser-0.62-23.el8.x86_64
sqlite-libs-3.26.0-13.el8.x86_64
iw-4.14-5.el8.x86_64
libpinyin-2.2.0-1.el8.x86_64
libxcrypt-4.1.1-4.el8.x86_64
libproxy-0.4.15-5.2.el8.x86_64
libsolv-0.7.16-2.el8.x86_64
freetype-2.9.1-4.el8_3.1.x86_64
soundtouch-2.0.0-3.el8.x86_64
samba-common-libs-4.13.3-3.el8.x86_64
yajl-2.1.0-10.el8.x86_64
libpaper-1.1.24-22.el8.x86_64
plymouth-0.9.4-9.20200615git1e36e30.el8.x86_64
libtdb-1.4.3-1.el8.x86_64
checkpolicy-2.9-1.el8.x86_64
cyrus-sasl-gssapi-2.1.27-5.el8.x86_64
libSM-1.2.3-1.el8.x86_64
fuse3-libs-3.2.1-12.el8.x86_64
gupnp-1.0.6-1.el8.x86_64
keyutils-libs-1.5.10-6.el8.x86_64
libndp-1.7-5.el8.x86_64
plymouth-graphics-libs-0.9.4-9.20200615git1e36e30.el8.x86_64
libwayland-server-1.17.0-1.el8.x86_64
pkgconf-1.4.2-1.el8.x86_64
python3-blockdev-2.24-5.el8.x86_64
libtevent-0.10.2-2.el8.x86_64
mallard-rng-1.0.3-1.el8.noarch
libblockdev-swap-2.24-5.el8.x86_64
grep-3.1-6.el8.x86_64
libsmbios-2.4.1-2.el8.x86_64
pipewire-0.3.6-1.el8.x86_64
libicu-60.3-2.el8_1.x86_64
libstemmer-0-10.585svn.el8.x86_64
pinentry-1.1.0-2.el8.x86_64
libxslt-1.1.32-6.el8.x86_64
memstrack-0.1.11-1.el8.x86_64
librepo-1.12.0-3.el8.x86_64
which-2.21-12.el8.x86_64
containernetworking-plugins-0.9.1-1.module+el8.4.0+10607+f4da7515.x86_64
python3-libdnf-0.55.0-7.el8.x86_64
libxcb-1.13.1-1.el8.x86_64
hyphen-2.8.8-9.el8.x86_64
libgxps-0.3.0-5.el8.x86_64
libXi-1.7.10-1.el8.x86_64
liba52-0.7.4-32.el8.x86_64
cronie-1.5.2-4.el8.x86_64
libXrandr-1.5.2-1.el8.x86_64
libdv-1.0.0-27.el8.x86_64
libstoragemgmt-1.8.7-1.el8.x86_64
orc-0.4.28-3.el8.x86_64
gavl-1.4.0-12.el8.x86_64
at-spi2-core-2.28.0-1.el8.x86_64
libsigc++20-2.10.0-6.el8.x86_64
libmcpp-2.7.2-20.el8.x86_64
libgee-0.20.1-1.el8.x86_64
libaio-0.3.112-1.el8.x86_64
libnet-1.1.6-15.el8.x86_64
fuse-overlayfs-1.4.0-2.module+el8.4.0+10607+f4da7515.x86_64
nettle-3.4.1-2.el8.x86_64
libvisual-0.4.0-25.el8.x86_64
tpm2-tss-2.3.2-3.el8.x86_64
libdrm-2.4.103-1.el8.x86_64
jq-1.5-12.el8.x86_64
python3-pytoml-0.1.14-5.git7dea353.el8.noarch
libwinpr-2.2.0-1.el8.x86_64
libasyncns-0.8-14.el8.x86_64
mobile-broadband-provider-info-20190618-2.el8.noarch
xfsprogs-5.0.0-8.el8.x86_64
findutils-4.6.0-20.el8.x86_64
kbd-legacy-2.0.4-10.el8.noarch
teamd-1.31-2.el8.x86_64
hunspell-en-US-0.20140811.1-12.el8.noarch
pciutils-3.7.0-1.el8.x86_64
python3-setuptools-39.2.0-6.el8.noarch
libtasn1-4.13-3.el8.x86_64
python3-libs-3.6.8-37.el8.x86_64
python3-pyxdg-0.25-16.el8.noarch
libxshmfence-1.3-2.el8.x86_64
libpcap-1.9.1-5.el8.x86_64
qemu-kvm-block-iscsi-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
libXt-1.1.5-12.el8.x86_64
platform-python-3.6.8-37.el8.x86_64
qemu-kvm-block-curl-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
bubblewrap-0.4.0-1.el8.x86_64
libarchive-3.3.3-1.el8.x86_64
bind-libs-9.11.26-3.el8.x86_64
gdbm-libs-1.18-1.el8.x86_64
openssl-1.1.1g-15.el8_3.x86_64
python3-simpleline-1.1.1-2.el8.noarch
libdvdread-5.0.3-9.el8.x86_64
gzip-1.9-12.el8.x86_64
libblockdev-nvdimm-2.24-5.el8.x86_64
libbytesize-1.4-3.el8.x86_64
krb5-libs-1.18.2-8.el8.x86_64
flatpak-libs-1.8.5-3.el8.x86_64
fuse-2.9.7-12.el8.x86_64
device-mapper-1.02.175-5.el8.x86_64
libreport-plugin-rhtsupport-2.9.5-15.el8.x86_64
libv4l-1.14.2-3.el8.x86_64
rpm-4.14.3-13.el8.x86_64
openssh-clients-8.0p1-5.el8.x86_64
cpio-2.12-10.el8.x86_64
libmount-2.32.1-27.el8.x86_64
usbmuxd-1.1.0-13.el8.x86_64
gsm-1.0.17-5.el8.x86_64
systemd-libs-239-45.el8.x86_64
xorg-x11-drv-fbdev-0.5.0-2.el8.x86_64
libraw1394-2.1.2-5.el8.x86_64
ca-certificates-2020.2.41-80.0.el8_2.noarch
python3-dmidecode-3.12.2-15.el8.x86_64
mesa-libEGL-20.3.3-2.el8.x86_64
rpm-libs-4.14.3-13.el8.x86_64
lvm2-2.03.11-5.el8.x86_64
speexdsp-1.2-0.13.rc3.el8.x86_64
trousers-lib-0.3.15-1.el8.x86_64
unbound-libs-1.7.3-15.el8.x86_64
libXmu-1.1.3-1.el8.x86_64
pam-1.3.1-14.el8.x86_64
crda-3.18_2020.04.29-1.el8.noarch
boost-random-1.66.0-10.el8.x86_64
os-prober-1.74-6.el8.x86_64
ModemManager-1.10.8-2.el8.x86_64
perl-Math-Complex-1.59-419.el8.noarch
net-tools-2.0-0.52.20160912git.el8.x86_64
xdg-utils-1.1.2-5.el8.noarch
python3-pycurl-7.43.0.2-4.el8.x86_64
irqbalance-1.4.0-6.el8.x86_64
glusterfs-api-6.0-49.1.el8.x86_64
open-vm-tools-11.2.0-2.el8.x86_64
tcpdump-4.9.3-1.el8.x86_64
python3-libsemanage-2.9-6.el8.x86_64
dhcp-libs-4.3.6-44.el8.x86_64
strace-5.7-2.el8.x86_64
container-selinux-2.158.0-1.module+el8.4.0+10607+f4da7515.noarch
kexec-tools-2.0.20-46.el8.x86_64
rhc-0.1.2-2.el8.x86_64
python3-firewall-0.8.2-6.el8.noarch
xdg-desktop-portal-gtk-1.6.0-1.el8.x86_64
langpacks-en-1.0-12.el8.noarch
python3-dasbus-1.2-2.el8.noarch
nss-3.53.1-17.el8_3.x86_64
ed-1.14.2-4.el8.x86_64
cups-filters-libs-1.20.0-24.el8.x86_64
librados2-12.2.7-9.el8.x86_64
hostname-3.20-6.el8.x86_64
libao-1.2.0-10.el8.x86_64
librbd1-12.2.7-9.el8.x86_64
google-noto-sans-sinhala-fonts-20161022-7.el8.1.noarch
speech-dispatcher-0.8.8-6.el8.x86_64
spice-glib-0.38-6.el8.x86_64
pnm2ppa-1.04-40.el8.x86_64
python3-linux-procfs-0.6.3-1.el8.noarch
tracker-miners-2.1.5-1.el8.x86_64
lsscsi-0.32-2.el8.x86_64
brltty-5.6-32.el8.x86_64
cups-filters-1.20.0-24.el8.x86_64
gnome-getting-started-docs-3.28.2-1.el8.noarch
pipewire0.2-libs-0.2.7-6.el8.x86_64
volume_key-libs-0.3.11-5.el8.x86_64
khmeros-base-fonts-5.0-25.el8.noarch
libmtp-1.1.14-3.el8.x86_64
nautilus-3.28.1-15.el8.x86_64
gnu-free-sans-fonts-20120503-18.el8.noarch
python3-gobject-3.28.3-2.el8.x86_64
udisks2-iscsi-2.9.0-6.el8.x86_64
julietaula-montserrat-fonts-7.200-2.el8.2.noarch
python3-pyatspi-2.26.0-6.el8.noarch
libcanberra-gtk3-0.30-18.el8.x86_64
lohit-gujarati-fonts-2.92.4-3.el8.noarch
gtk3-3.22.30-6.el8.x86_64
metacity-3.28.0-1.el8.x86_64
lohit-tamil-fonts-2.91.3-3.el8.noarch
clutter-1.26.2-8.el8.x86_64
gnome-settings-daemon-3.32.0-14.el8.x86_64
sil-abyssinica-fonts-1.200-13.el8.noarch
gnome-online-accounts-3.28.2-2.el8.x86_64
libvirt-daemon-driver-storage-core-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
gutenprint-doc-5.2.14-3.el8.x86_64
gjs-1.56.2-5.el8.x86_64
libvirt-daemon-driver-storage-iscsi-direct-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
libertas-usb8388-firmware-20201218-102.git05789708.el8.noarch
libgnomekbd-3.26.0-4.el8.x86_64
libvirt-daemon-driver-storage-scsi-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
iwl6000g2a-firmware-18.168.6.1-102.el8.1.noarch
nm-connection-editor-1.18.0-2.el8.x86_64
python3-ethtool-0.14-3.el8.x86_64
iwl3160-firmware-25.30.13.0-102.el8.1.noarch
pangomm-2.40.1-6.el8.x86_64
python3-dnf-4.4.2-11.el8.noarch
iwl105-firmware-18.168.6.1-102.el8.1.noarch
yelp-3.28.1-3.el8.x86_64
kpatch-dnf-0.2-3.el8.noarch
NetworkManager-config-server-1.30.0-7.el8.noarch
gnome-desktop3-3.32.2-1.el8.x86_64
python3-magic-5.33-16.el8_3.1.noarch
apr-util-openssl-1.6.1-6.el8.x86_64
openssh-askpass-8.0p1-5.el8.x86_64
bind-utils-9.11.26-3.el8.x86_64
git-core-doc-2.27.0-1.el8.noarch
gnome-keyring-3.28.2-1.el8.x86_64
python3-urllib3-1.24.2-5.el8.noarch
libdmapsharing-2.9.37-5.el8.x86_64
ibus-gtk3-1.5.19-12.el8.x86_64
gdm-3.28.3-39.el8.x86_64
libreport-gtk-2.9.5-15.el8.x86_64
gnome-shell-extension-desktop-icons-3.32.1-14.el8.noarch
perl-Error-0.17025-2.el8.noarch
plymouth-plugin-label-0.9.4-9.20200615git1e36e30.el8.x86_64
gnome-shell-extension-window-list-3.32.1-14.el8.noarch
httpd-filesystem-2.4.37-39.module+el8.4.0+9658+b87b2deb.noarch
urw-base35-bookman-fonts-20170801-10.el8.noarch
python3-schedutils-0.6-6.el8.x86_64
python3-pycparser-2.14-14.el8.noarch
urw-base35-nimbus-mono-ps-fonts-20170801-10.el8.noarch
authselect-1.2.2-2.el8.x86_64
sshpass-1.06-9.el8.x86_64
urw-base35-standard-symbols-ps-fonts-20170801-10.el8.noarch
python3-blivet-3.2.2-10.el8.noarch
ansible-2.9.18-2.el8.noarch
ghostscript-9.27-1.el8.x86_64
python3-pid-2.1.1-7.el8.noarch
gnutls-3.6.14-9.el8.x86_64
libgsf-1.14.41-5.el8.x86_64
sos-4.0-11.el8.noarch
virt-manager-common-2.2.1-4.el8.noarch
libvirt-libs-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
python3-pytz-2017.2-9.el8.noarch
libvirt-client-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
libgphoto2-2.5.16-3.el8.x86_64
initial-setup-0.3.81.7-1.el8.x86_64
sane-backends-drivers-scanners-1.0.27-22.el8.x86_64
python3-html5lib-0.999999999-6.el8.noarch
numad-0.5-26.20150602git.el8.x86_64
libibumad-32.0-4.el8.x86_64
dnsmasq-2.79-15.el8.x86_64
gnome-boxes-3.36.5-8.el8.x86_64
hypervvssd-0-0.30.20180415git.el8.x86_64
gnome-classic-session-3.32.1-14.el8.noarch
librelp-1.2.16-1.el8.x86_64
cockpit-podman-29-2.module+el8.4.0+10607+f4da7515.noarch
libipa_hbac-2.4.0-9.el8.x86_64
dnf-plugins-core-4.0.18-4.el8.noarch
podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64
gom-0.4-1.el8.x86_64
gnome-disk-utility-3.28.3-2.el8.x86_64
totem-pl-parser-3.26.1-2.el8.x86_64
evince-nautilus-3.28.4-11.el8.x86_64
dracut-squash-049-135.git20210121.el8.x86_64
gvfs-archive-1.36.2-11.el8.x86_64
sssd-common-pac-2.4.0-9.el8.x86_64
gvfs-smb-1.36.2-11.el8.x86_64
sudo-1.8.29-7.el8.x86_64
xorg-x11-drv-wacom-0.38.0-1.el8.x86_64
luksmeta-9-4.el8.x86_64
vim-enhanced-8.0.1763-15.el8.x86_64
isns-utils-libs-0.99-1.el8.x86_64
gnome-remote-desktop-0.1.8-3.el8.x86_64
gstreamer1-plugins-bad-free-1.16.1-1.el8.x86_64
plymouth-system-theme-0.9.4-9.20200615git1e36e30.el8.x86_64
libvirt-daemon-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
ibus-libpinyin-1.10.0-2.el8.x86_64
libvirt-daemon-driver-nodedev-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
gnome-font-viewer-3.28.0-1.el8.x86_64
perl-Digest-1.17-395.el8.noarch
gnome-characters-3.28.2-1.el8.x86_64
rpm-ostree-libs-2020.7-3.el8.x86_64
perl-URI-1.73-3.el8.noarch
file-roller-3.28.1-3.el8.x86_64
osbuild-selinux-28-1.20210408giteb74ddf.20210408giteb74ddf.el8.noarch
perl-IO-Socket-IP-0.39-5.el8.noarch
xdg-user-dirs-gtk-0.10-13.el8.x86_64
osbuild-composer-core-30-1.20210617git2ed4dde.el8.x86_64
perl-Term-Cap-1.17-395.el8.noarch
pinfo-0.6.10-18.el8.x86_64
perl-podlators-4.11-1.el8.noarch
yelp-tools-3.28.0-3.el8.noarch
perl-MIME-Base64-3.15-396.el8.x86_64
xorg-x11-drv-nouveau-1.0.15-4.el8.1.x86_64
perl-Socket-2.027-3.el8.x86_64
PackageKit-command-not-found-1.1.12-6.el8.x86_64
perl-libs-5.26.3-419.el8.x86_64
openssh-server-8.0p1-5.el8.x86_64
perl-Text-Tabs+Wrap-2013.0523-395.el8.noarch
NetworkManager-adsl-1.30.0-7.el8.x86_64
perl-PathTools-3.74-1.el8.x86_64
usbutils-010-3.el8.x86_64
perl-interpreter-5.26.3-419.el8.x86_64
shared-mime-info-1.9-3.el8.x86_64
psacct-6.6.3-4.el8.x86_64
hwdata-0.314-8.8.el8.noarch
redhat-backgrounds-84.4-1.el8.noarch
libnftnl-1.1.5-4.el8.x86_64
xmlsec1-openssl-1.2.25-4.el8.x86_64
fontconfig-2.13.1-3.el8.x86_64
ledmon-0.95-1.el8.x86_64
gnome-control-center-filesystem-3.28.2-27.el8.noarch
python3-setools-4.3.0-2.el8.x86_64
unzip-6.0-44.el8.x86_64
dracut-network-049-135.git20210121.el8.x86_64
libsecret-0.18.6-1.el8.x86_64
xorg-x11-utils-7.5-28.el8.x86_64
geolite2-city-20180605-1.el8.noarch
libblockdev-dm-2.24-5.el8.x86_64
libksba-1.3.5-7.el8.x86_64
nss-sysinit-3.53.1-17.el8_3.x86_64
polkit-0.115-11.el8.x86_64
time-1.9-3.el8.x86_64
dnf-data-4.4.2-11.el8.noarch
cups-ipptool-2.2.6-38.el8.x86_64
webrtc-audio-processing-0.3-9.el8.x86_64
evince-libs-3.28.4-11.el8.x86_64
libldb-2.2.0-2.el8.x86_64
google-noto-sans-tagalog-fonts-20161022-7.el8.1.noarch
google-noto-emoji-fonts-20180508-4.el8.noarch
python3-speechd-0.8.8-6.el8.x86_64
libpath_utils-0.2.1-39.el8.x86_64
evince-3.28.4-11.el8.x86_64
PackageKit-glib-1.1.12-6.el8.x86_64
mtr-0.92-3.el8.x86_64
tigervnc-license-1.11.0-6.el8.noarch
python3-brlapi-0.6.7-32.el8.x86_64
lame-libs-3.100-6.el8.x86_64
fprintd-1.90.9-2.el8.x86_64
desktop-file-utils-0.23-8.el8.x86_64
google-noto-sans-cjk-ttc-fonts-20190416-1.el8.noarch
libwacom-data-1.6-2.el8.noarch
frei0r-plugins-1.6.1-7.el8.x86_64
libteam-1.31-2.el8.x86_64
fwupd-1.5.5-3.el8.x86_64
libgusb-0.3.0-1.el8.x86_64
lohit-assamese-fonts-2.91.5-3.el8.noarch
edk2-ovmf-20200602gitca407c7246bf-4.el8.noarch
gd-2.2.5-7.el8.x86_64
libcdio-paranoia-10.2+0.94+2-3.el8.x86_64
gnome-bluetooth-3.34.3-1.el8.x86_64
libevent-2.1.8-5.el8.x86_64
lohit-telugu-fonts-2.5.5-3.el8.noarch
redhat-release-8.4-0.6.el8.x86_64
gtk2-2.24.32-5.el8.x86_64
libusbmuxd-1.0.10-9.el8.x86_64
nfs-utils-2.3.3-41.el8.x86_64
libinput-1.16.3-1.el8.x86_64
words-3.0-28.el8.noarch
pkgconf-m4-1.4.2-1.el8.noarch
libgweather-3.28.2-3.el8.x86_64
libxkbcommon-x11-0.9.1-1.el8.x86_64
libvirt-daemon-driver-storage-rbd-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
glib-networking-2.56.1-1.1.el8.x86_64
iwl6000-firmware-9.221.4.1-102.el8.1.noarch
bash-4.4.19-14.el8.x86_64
gstreamer1-plugins-good-1.16.1-2.el8.x86_64
lua-lpeg-1.0.1-6.el8.x86_64
python3-libcomps-0.1.11-5.el8.x86_64
python3-dateutil-2.6.1-6.el8.noarch
iwl1000-firmware-39.31.5.1-102.el8.1.noarch
audit-libs-3.0-0.17.20191104git1c2f876.el8.x86_64
libtimezonemap-0.4.5.1-3.el8.x86_64
aspell-0.60.6.1-21.el8.x86_64
python3-meh-gui-0.47.2-1.el8.noarch
libpinyin-data-2.2.0-1.el8.x86_64
apr-util-1.6.1-6.el8.x86_64
info-6.5-6.el8.x86_64
pinentry-gtk-1.1.0-2.el8.x86_64
graphite2-1.3.10-10.el8.x86_64
python3-pysocks-1.6.8-3.el8.noarch
libwbclient-4.13.3-3.el8.x86_64
python3-pyasn1-0.3.7-6.el8.noarch
readline-7.0-10.el8.x86_64
spice-server-0.14.3-4.el8.x86_64
vim-common-8.0.1763-15.el8.x86_64
gnome-control-center-3.28.2-27.el8.x86_64
xorg-x11-xinit-1.3.4-18.el8.x86_64
python3-babel-2.5.1-5.el8.noarch
libX11-xcb-1.6.8-4.el8.x86_64
libwnck3-3.24.1-2.el8.x86_64
hardlink-1.3-6.el8.x86_64
gnome-shell-extension-places-menu-3.32.1-14.el8.noarch
libreport-plugin-reportuploader-2.9.5-15.el8.x86_64
redhat-logos-httpd-84.4-1.el8.noarch
libwayland-client-1.17.0-1.el8.x86_64
urw-base35-c059-fonts-20170801-10.el8.noarch
pkgconf-pkg-config-1.4.2-1.el8.x86_64
realmd-0.16.3-22.el8.x86_64
libblockdev-part-2.24-5.el8.x86_64
libsodium-1.0.18-2.el8.x86_64
pcre-8.42-4.el8.x86_64
urw-base35-z003-fonts-20170801-10.el8.noarch
libsss_autofs-2.4.0-9.el8.x86_64
python3-ntplib-0.3.3-10.el8.noarch
python3-pyudev-0.21.0-7.el8.noarch
gnutls-dane-3.6.14-9.el8.x86_64
libref_array-0.1.5-39.el8.x86_64
libmediaart-1.9.4-4.el8.x86_64
npth-1.5-4.el8.x86_64
cockpit-storaged-238.2-1.el8.noarch
ostree-libs-2020.7-4.el8.x86_64
virt-install-2.2.1-4.el8.noarch
libXau-1.0.9-3.el8.x86_64
sane-backends-drivers-cameras-1.0.27-22.el8.x86_64
jbig2dec-libs-0.16-1.el8.x86_64
python3-webencodings-0.5.1-6.el8.noarch
crontabs-1.11-17.20190603git.el8.noarch
libXrender-0.9.10-7.el8.x86_64
systemd-container-239-45.el8.x86_64
libestr-0.1.10-1.el8.x86_64
libvirt-daemon-kvm-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
switcheroo-control-1.1-5.el8.x86_64
libassuan-2.5.1-3.el8.x86_64
oddjob-0.34.7-1.el8.x86_64
mcpp-2.7.2-20.el8.x86_64
system-config-printer-udev-1.5.11-13.el8.x86_64
virt-what-1.18-6.el8.x86_64
sed-4.5-2.el8.x86_64
conmon-2.0.26-1.module+el8.4.0+10607+f4da7515.x86_64
mpg123-libs-1.25.10-2.el8.x86_64
gnome-color-manager-3.28.0-3.el8.x86_64
libjose-10-2.el8.x86_64
lz4-libs-1.8.3-2.el8.x86_64
libslirp-4.3.1-1.module+el8.4.0+10607+f4da7515.x86_64
linux-firmware-20201218-102.git05789708.el8.noarch
gvfs-afp-1.36.2-11.el8.x86_64
genisoimage-1.1.11-39.el8.x86_64
composer-cli-28.14.58-1.el8.x86_64
hunspell-en-0.20140811.1-12.el8.noarch
sssd-krb5-2.4.0-9.el8.x86_64
cyrus-sasl-lib-2.1.27-5.el8.x86_64
firefox-78.9.0-1.el8_3.x86_64
python3-ordered-set-2.0.2-4.el8.noarch
libexif-0.6.22-5.el8_3.x86_64
clevis-luks-15-1.el8.x86_64
iptables-libs-1.8.4-17.el8.x86_64
ibus-typing-booster-2.1.0-5.el8.noarch
libblockdev-mpath-2.24-5.el8.x86_64
mpfr-3.1.6-1.el8.x86_64
neon-0.30.2-6.el8.x86_64
libdb-utils-5.3.28-40.el8.x86_64
ibus-kkc-1.5.22-9.el8.x86_64
python3-subscription-manager-rhsm-1.28.13-2.el8.x86_64
libXdmcp-1.1.3-1.el8.x86_64
libvirt-daemon-driver-nwfilter-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
cracklib-2.9.6-15.el8.x86_64
gnome-calculator-3.28.2-1.el8.x86_64
ostree-2020.7-4.el8.x86_64
rpm-ostree-2020.7-3.el8.x86_64
libXxf86vm-1.1.4-9.el8.x86_64
perl-Net-SSLeay-1.88-1.module+el8.3.0+6446+594cad75.x86_64
elfutils-debuginfod-client-0.182-3.el8.x86_64
mousetweaks-3.12.0-11.el8.x86_64
adcli-0.8.2-9.el8.x86_64
osbuild-composer-30-1.20210617git2ed4dde.el8.x86_64
libselinux-utils-2.9-5.el8.x86_64
perl-File-Temp-0.230.600-1.el8.noarch
dbus-libs-1.12.8-12.el8.x86_64
kernel-4.18.0-304.el8.x86_64
libqmi-utils-1.24.0-1.el8.x86_64
libieee1284-0.2.11-28.el8.x86_64
perl-Storable-3.11-3.el8.x86_64
openssl-libs-1.1.1g-15.el8_3.x86_64
xfsdump-3.1.8-2.el8.x86_64
lvm2-libs-2.03.11-5.el8.x86_64
sbc-1.3-9.el8.x86_64
perl-Scalar-List-Utils-1.49-2.el8.x86_64
libutempter-1.1.6-14.el8.x86_64
PackageKit-gstreamer-plugin-1.1.12-6.el8.x86_64
bluez-obexd-5.52-4.el8.x86_64
boost-chrono-1.66.0-10.el8.x86_64
perl-constant-1.33-396.el8.noarch
gettext-0.19.8.1-17.el8.x86_64
microcode_ctl-20210216-1.el8.x86_64
tigervnc-server-minimal-1.11.0-6.el8.x86_64
perl-DBI-1.641-3.module+el8.3.0+6481+fbe55708.x86_64
xcb-util-0.4.0-10.el8.x86_64
systemd-239-45.el8.x86_64
dejavu-fonts-common-2.35-7.el8.noarch
dracut-config-rescue-049-135.git20210121.el8.x86_64
glusterfs-cli-6.0-49.1.el8.x86_64
gnome-session-wayland-session-3.28.1-10.el8.x86_64
lua-5.3.4-11.el8.x86_64
json-glib-1.4.4-1.el8.x86_64
libreport-filesystem-2.9.5-15.el8.x86_64
blktrace-1.2.0-10.el8.x86_64
python3-slip-0.6.4-11.el8.noarch
flatpak-1.8.5-3.el8.x86_64
libnl3-cli-3.5.0-1.el8.x86_64
gobject-introspection-1.56.1-1.el8.x86_64
python3-setuptools-wheel-39.2.0-6.el8.noarch
nano-2.9.8-1.el8.x86_64
timedatex-0.5-3.el8.x86_64
grilo-0.3.6-2.el8.x86_64
boost-date-time-1.66.0-10.el8.x86_64
ibus-libs-1.5.19-12.el8.x86_64
dejavu-sans-fonts-2.35-7.el8.noarch
google-noto-sans-mandaic-fonts-20161022-7.el8.1.noarch
espeak-ng-1.49.2-4.el8.x86_64
evolution-data-server-3.28.5-15.el8.x86_64
libedit-3.1-23.20170329cvs.el8.x86_64
libreport-2.9.5-15.el8.x86_64
thai-scalable-fonts-common-0.6.5-1.el8.noarch
dos2unix-7.4.0-3.el8.x86_64
python3-productmd-1.11-3.el8.noarch
grilo-plugins-0.3.8-1.el8.x86_64
userspace-rcu-0.10.1-4.el8.x86_64
policycoreutils-2.9-14.el8.x86_64
seavgabios-bin-1.13.0-2.module+el8.3.0+7353+9de0a3cc.noarch
thai-scalable-waree-fonts-0.6.5-1.el8.noarch
usb_modeswitch-2.5.2-1.el8.x86_64
udisks2-2.9.0-6.el8.x86_64
libshout-2.2.2-19.el8.x86_64
dbus-x11-1.12.8-12.el8.x86_64
libX11-common-1.6.8-4.el8.noarch
dejavu-serif-fonts-2.35-7.el8.noarch
setroubleshoot-plugins-3.3.13-1.el8.noarch
sound-theme-freedesktop-0.8-9.el8.noarch
glx-utils-8.4.0-5.20181118git1830dcb.el8.x86_64
kernel-core-4.18.0-304.el8.x86_64
anaconda-user-help-8.3.3-1.el8.noarch
lohit-kannada-fonts-2.5.4-3.el8.noarch
gcr-3.28.0-1.el8.x86_64
color-filesystem-1-20.el8.noarch
libXvMC-1.0.12-1.el8.x86_64
device-mapper-event-libs-1.02.175-5.el8.x86_64
urw-base35-fonts-common-20170801-10.el8.noarch
sil-padauk-fonts-3.003-1.el8.noarch
clutter-gst3-3.0.26-1.el8.x86_64
libvirt-daemon-driver-storage-gluster-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
libXxf86dga-1.1.5-1.el8.x86_64
rpm-plugin-selinux-4.14.3-13.el8.x86_64
libselinux-2.9-5.el8.x86_64
iwl6050-firmware-41.28.5.1-102.el8.1.noarch
vte291-0.52.4-2.el8.x86_64
kernel-tools-4.18.0-304.el8.x86_64
nss-softokn-3.53.1-17.el8_3.x86_64
geoclue2-2.5.5-2.el8.x86_64
libstdc++-8.4.1-1.el8.x86_64
iwl2000-firmware-18.168.6.1-102.el8.1.noarch
gnome-terminal-3.28.3-3.el8.x86_64
python3-dnf-plugins-core-4.0.18-4.el8.noarch
jasper-libs-2.0.14-4.el8.x86_64
upower-0.99.7-3.el8.x86_64
libxml2-2.9.7-9.el8.x86_64
apr-1.6.3-11.el8.x86_64
gnome-session-xsession-3.28.1-10.el8.x86_64
python3-ply-3.9-9.el8.noarch
libconfig-1.5-9.el8.x86_64
libxklavier-5.4-11.el8.x86_64
libjpeg-turbo-1.5.3-10.el8.x86_64
gstreamer1-plugins-ugly-free-1.16.1-1.el8.x86_64
subscription-manager-1.28.13-2.el8.x86_64
kernel-tools-libs-4.18.0-304.el8.x86_64
libsmbclient-4.13.3-3.el8.x86_64
elfutils-libelf-0.182-3.el8.x86_64
ibus-1.5.19-12.el8.x86_64
gnome-shell-extension-common-3.32.1-14.el8.noarch
efivar-libs-37-4.el8.x86_64
python3-syspurpose-1.28.13-2.el8.x86_64
libxkbcommon-0.9.1-1.el8.x86_64
git-2.27.0-1.el8.x86_64
plymouth-theme-spinner-0.9.4-9.20200615git1e36e30.el8.x86_64
python3-requests-file-1.4.3-5.el8.noarch
libpipeline-1.5.0-2.el8.x86_64
iptables-ebtables-1.8.4-17.el8.x86_64
lcms2-2.9-2.el8.x86_64
python3-cryptography-3.2.1-4.el8.x86_64
urw-base35-nimbus-sans-fonts-20170801-10.el8.noarch
fprintd-pam-1.90.9-2.el8.x86_64
gawk-4.2.1-2.el8.x86_64
rtkit-0.11-19.el8.x86_64
pixman-0.38.4-1.el8.x86_64
gvnc-0.9.0-2.el8.x86_64
python3-ptyprocess-0.5.2-4.el8.noarch
lz4-1.8.3-2.el8.x86_64
gnupg2-2.2.20-2.el8.x86_64
jansson-2.11-3.el8.x86_64
autogen-libopts-5.18.12-8.el8.x86_64
libvirt-gobject-3.0.0-1.el8.x86_64
anaconda-core-33.16.4.15-1.el8.x86_64
giflib-5.1.4-3.el8.x86_64
usbredir-0.8.0-1.el8.x86_64
libXext-1.3.4-1.el8.x86_64
mdadm-4.1-15.el8.x86_64
python3-tracer-0.7.5-2.el8.noarch
libthai-0.1.27-2.el8.x86_64
initscripts-10.00.15-1.el8.x86_64
libXinerama-1.1.4-1.el8.x86_64
hypervfcopyd-0-0.30.20180415git.el8.x86_64
orca-3.28.2-1.el8.noarch
libijs-0.35-5.el8.x86_64
libxmlb-0.1.15-1.el8.x86_64
libtar-1.2.20-15.el8.x86_64
qpdf-libs-7.1.1-10.el8.x86_64
insights-client-3.1.1-1.el8_3.noarch
runc-1.0.0-70.rc92.module+el8.4.0+10607+f4da7515.x86_64
ndctl-libs-71.1-2.el8.x86_64
libidn2-2.2.0-1.el8.x86_64
dleyna-server-0.6.0-3.el8.x86_64
gnome-software-3.36.1-5.el8.x86_64
twolame-libs-0.3.13-12.el8.x86_64
qemu-kvm-common-4.2.0-48.module+el8.4.0+10368+630e803b.x86_64
xz-5.2.4-3.el8.x86_64
sssd-common-2.4.0-9.el8.x86_64
gvfs-gphoto2-1.36.2-11.el8.x86_64
dhcp-common-4.3.6-44.el8.noarch
libnfsidmap-2.3.3-41.el8.x86_64
libplist-2.0.0-10.el8.x86_64
python3-pwquality-1.4.4-3.el8.x86_64
rasdaemon-0.6.1-5.el8.x86_64
rdma-core-32.0-4.el8.x86_64
librdmacm-32.0-4.el8.x86_64
file-5.33-16.el8_3.1.x86_64
iscsi-initiator-utils-6.2.1.2-1.gita8fcb37.el8.x86_64
hyperv-daemons-0-0.30.20180415git.el8.x86_64
libkcapi-1.2.0-2.el8.x86_64
freerdp-libs-2.2.0-1.el8.x86_64
bluez-libs-5.52-4.el8.x86_64
libvirt-daemon-config-network-6.0.0-35.module+el8.4.0+10230+7a9b21e4.x86_64
ibus-m17n-1.3.4-26.el8.x86_64
crypto-policies-20210209-1.gitbfb6bed.el8_3.noarch
clevis-15-1.el8.x86_64
opus-1.3-0.4.beta.el8.x86_64
perl-Data-Dumper-2.167-399.el8.x86_64
gtk2-immodule-xim-2.24.32-5.el8.x86_64
libnsl2-1.2.0-2.20180605git4a062cf.el8.x86_64
python3-gpg-1.13.1-7.el8.x86_64
exiv2-0.27.3-2.el8.x86_64
osbuild-ostree-28-1.20210408giteb74ddf.20210408giteb74ddf.el8.noarch
perl-IO-Socket-SSL-2.066-4.module+el8.3.0+6446+594cad75.noarch
pulseaudio-module-x11-14.0-2.el8.x86_64
libcroco-0.6.12-4.el8_2.1.x86_64
passwd-0.80-3.el8.x86_64
psmisc-23.1-5.el8.x86_64
perl-Text-ParseWords-3.30-395.el8.noarch
xorg-x11-drv-evdev-2.10.6-2.el8.x86_64
libblkid-2.32.1-27.el8.x86_64
xorg-x11-drv-vesa-2.4.0-3.el8.x86_64
mesa-libglapi-20.3.3-2.el8.x86_64
perl-Carp-1.42-396.el8.noarch
rsyslog-gnutls-8.1911.0-7.el8.x86_64
cryptsetup-libs-2.3.3-4.el8.x86_64
libverto-libevent-0.3.0-5.el8.x86_64
mesa-libGL-20.3.3-2.el8.x86_64
perl-File-Path-2.15-2.el8.noarch
alsa-ucm-1.2.4-5.el8.noarch
systemd-pam-239-45.el8.x86_64
kmod-kvdo-6.2.4.26-77.el8.x86_64
libpsl-0.20.2-6.el8.x86_64
achilleas-k commented 3 years ago

Some more info on this from me:

I came across this issue while testing #193 through Image Builder. At first I ignored it, but then it would happen reliably that I made a note to look into it. Yi also started coming across it in his own testing and suggested opening an issue for tracking.

I haven't had the opportunity to look into it further, so I don't know what kind of information would be relevant. It doesn't seem to occur when I run composer without Image Builder, which is strange because when I do, I run IB in a container and Composer in a separate VM. I asked for Yi to post the list of installed packages on his composer/worker host hoping that might help reproduce.

teg commented 3 years ago

Sounds to me that an RPM post-install script is triggering a device successfully (writing to the uevent file in /sys), but is not getting the corresponding uevent back, possibly because we are not in the root (network?) namespace.

@gicmo, @dvdhrm do you have any ideas (if that is in fact the case, and if so what to do...)?

gicmo commented 3 years ago

@achilleas-k has seen that locally. There the executed command line hinted towards rng-tools

dvdhrm commented 3 years ago

udevadm trigger --sysname-match=hw_random --settle

This is very likely wrong to call in an RPM scriptlet. Scriptlets cannot assume to run on the machine they are installed for. Cross-install etc. breaks then. Furthermore, I do not see the point of triggering the hardware-random generators and waiting for them. This smells like some over-zealous crypto caller making sure the hw-random generators are actually available and used.

The --settle part can definitely break if you call udevadm on a normal /sys mount but are not in the correct network-namespace. You will never see the uevents you triggered. osbuild runs in such a setup, iirc. I can imagine that the scriptlet or some other part has a guard to check whether they run in a container first (systemd has the in_container() check, for instance, so udevadm might have it as well), and hence might behave differently in some setups.

yih-redhat commented 3 years ago

Update on this issue:

I set up a new test machine, a RHEL84 vm, and installed the latest main branch of osbuild-composer and image-builder.

This time, image-builder can build commit tar image successfully, and if create a kickstart file and install Edge vm with this commit tar, everything is fine in the Edge vm. I can login with the user/pass or ssh key defined in kickstart file.

For installer ISO, still blocked by 'udevadm' issue. after killed udevadm process, imge-builder can build install ISO image successfully. But something is wrong with this installer ISO image, if use kickstart file to install Edge vm with this installer ISO file, can NOT login Edge vm with the user/pass or ssh key defined in kickstart file.

I am afraid this issue is going to be a blocker issue for Edge APIs in image-builder service, that image-builder service cannot build installer ISO image correctly. @gicmo @teg @achilleas-k @dvdhrm

Reproduce steps:

  1. Send request to build a commit tar image. curl -w '%{http_code}' -H 'x-rh-identity: ${identity}' -H 'Content-Type: application/json' --request POST --data @/usr/share/tests/image-builder/edge/commit_body.json http://localhost:8086/api/image-builder/v1.0/compose

Request body is:

{
    "distribution": "rhel-84",
    "customizations": {
      "packages": [
        "python36"
      ]
    },
    "ostree": {
      "ref": "rhel/8/x86_64/edge"
    },
    "image_requests": [
      {
        "architecture": "x86_64",
        "image_type": "rhel-edge-commit",
        "upload_request": {
          "type": "aws.s3",
          "options": {}
        }
      }
    ]
}
  1. Image-builder can build this image and upload to s3 successfully.
  2. Download and extract commit tar and serve over httpd.
  3. Create a kickstart file, define user/password and ssh key in it. Install Edge vm with the commit repo.
  4. Can install Edge vm successfully, can login/ssh to it, run some sanity test, everything is fine.
  5. Send request to build an installer ISO image. curl -w '%{http_code}' -H 'x-rh-identity: ${identity}' -H 'Content-Type: application/json' --request POST --data @/usr/share/tests/image-builder/edge/installer_body.json http://localhost:8086/api/image-builder/v1.0/compose

Request body is:

{
    "distribution": "rhel-84",
    "customizations": {
      "packages": [
        "python36"
      ]
    },
    "image_requests": [
      {
        "architecture": "x86_64",
        "image_type": "rhel-edge-installer",
        "ostree": {
          "ref": "rhel/8/x86_64/edge",
          "url": "http://192.168.100.1/repo"
        },
        "upload_request": {
          "type": "aws.s3",
          "options": {}
        }
      }
    ]
}

  1. osbuild-worker hang at "udevadm" step, and cannot proceed.

    [root@rhel84iso2 edge]# pgrep osbuild-worker 
305891
[root@rhel84iso2 edge]# pstree 305891
osbuild-worker─┬─osbuild─┬─bwrap───bwrap───org.osbuild.rhe───org.osbuild.rpm───rpm───sh───udevadm
           │         └─4*[{osbuild}]
           └─8*[{osbuild-worker}]

  2. kill udevadm and the build process continues:

    [root@rhel84iso2 edge]# pstree 305891
osbuild-worker─┬─osbuild─┬─bwrap───bwrap───org.osbuild.rhe───org.osbuild.rpm───rpm───sh───udevadm
           │         └─4*[{osbuild}]
           └─8*[{osbuild-worker}]
[root@rhel84iso2 edge]# pkill udevadm
[root@rhel84iso2 edge]# pstree 305891
osbuild-worker─┬─osbuild─┬─bwrap───bwrap───org.osbuild.rhe───org.osbuild.rpm───rpm
           │         └─4*[{osbuild}]
           └─8*[{osbuild-worker}]

  3. Image-builder can build ISO image and upload to s3 successfully.

  4. Download installer ISO file.

  5. Create a kickstart file, define user/pass and ssh key, then install Edge vm with installer ISO file

  6. Can install Edge vm successfully, but can NOT login with user/pass or ssh key.

gicmo commented 3 years ago

I had a quick second look and indeed the rngd-wake-threshold.service from rng-tools contains the line:

ExecStart=/bin/sh -c "PSIZE=$(cat /proc/sys/kernel/random/poolsize); let THRESH=$PSIZE*3/4; echo $THRESH>/proc/sys/kernel/random/write_wakeup_threshold; udevadm trigger --sysname-match=hw_random --settle"

Which is excluded in most artefacts, but explicitly included in the installer itself: e.g. rhel84/distro.go:1439. Might have been a mistake to include it. Should be easy to remove it and test if the installer still works as expected. The package list comes from the Lorax templates runtime-install.tmpl:L134. @bcl added it via 690feba 6 years ago. These days it might not be necessary (see rhbz#1778762 and rhbz#1830280), hence why we remove it from other artefacts. Might still be worth filing an issue against the rng-tools so we don't trigger that during rpm, since a user could include the rng-tools in a blueprint and add it back and thus triggering that bug again.

bcl commented 3 years ago

FWIW I think that was added as part of the change to anaconda that waits for the entropy pool to fill. I'm pretty sure none of that was ever really needed -- from my reading about the rng the pool, even when low, is still producing suitable data. But the code still exists in Anaconda.

teg commented 3 years ago

I wonder if we can somehow fix this in osbuild. Can we stop the device (or all real hardware) from appearing in /sys? Or can uevent be made read only? Even if we fix the immediate problem, this seems like it could easily reappear.

Besides, RHEL8.4 has already been released, seems unlikely we will get this fixed now.

gicmo commented 3 years ago

I filed rhbz#1975554 against rig-tools.

henrywang commented 3 years ago

@gicmo @teg Do you have any ideas about how our CI and my downstream CI can't catch this issue? Does our test miss something? Thanks.

dvdhrm commented 3 years ago

I wonder if we can somehow fix this in osbuild. Can we stop the device (or all real hardware) from appearing in /sys? Or can uevent be made read only? Even if we fix the immediate problem, this seems like it could easily reappear.

Just checked: If you mount /sys read-only, then udevadm will skip the trigger operation and return immediately.

teg commented 3 years ago

I wonder if we can somehow fix this in osbuild. Can we stop the device (or all real hardware) from appearing in /sys? Or can uevent be made read only? Even if we fix the immediate problem, this seems like it could easily reappear.

Just checked: If you mount /sys read-only, then udevadm will skip the trigger operation and return immediately.

Probably the right thing to do regardless, no? Or are there namespaced things in there we actually want/need rw access to?

gicmo commented 3 years ago

I had a look at the udevadm source and since v241 it includes a check if it is running inside a chroot: https://github.com/systemd/systemd/commit/c494b739a47359ab2697482f52545e2a6d1c86ad. RHEL 8 is on v239 though :/

I confirmed and with a read only /sys (which apparently also what docker does, unless you run a privileged container) you get:

bash-5.1# udevadm trigger --sysname-match=hw_random --settle
hw_random: Failed to write 'change' to '/sys/devices/virtual/misc/hw_random/uevent': Read-only file system
bash-5.1# echo $?
1
nefigtut commented 3 years ago

hello, thank you for raising this issue, i'll fix rngd-wake-threshold.service in rng-tools, the related rhbz is mentioned above.

yih-redhat commented 3 years ago

@gicmo @achilleas-k @henrywang With gicmo's PR https://github.com/osbuild/osbuild/pull/691, the udevadm hang issue has been fixed. Now I can call image-builder Rest API to build installer ISO image successfully.