search

openshift / os

89 stars 105 forks source link

Add initial C10S variant #1498

Open travier opened 2 months ago

travier commented 2 months ago

c9s: Point image-c9s to image-rhel-9.4

c9s: Use RPMs from 9.4 RHAOS repo

Revert "c9s.repo: temporarily use mirrored repos"

This reverts commit 88e41a0fef0dbe8e0b2d103cadd37217c8cd0933.

DoNotMerge: CI changes to test C10S builds

manifests: Add initial c10s based variant

See: https://github.com/openshift/os/issues/1466

kola-denylist.yaml: Update for c10s variant

c10s.repo: Skip GPG checks for now

Corresponding Fedora CoreOS changes: https://github.com/coreos/fedora-coreos-config/pull/3015

travier commented 2 months ago

Do not merge as it includes changes that we do not want but that are here to test in CI (see commits). We'll have to update the Prow CI config once ready. /hold

openshift-ci[bot] commented 2 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: travier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/os/blob/master/OWNERS)~~ [travier] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
travier commented 2 months ago

/retest

travier commented 2 months ago

/retest

travier commented 1 month ago

/retest

jlebon commented 1 month ago

Hmm, almost seems like the logs are truncated... This is probably fallout from https://github.com/coreos/coreos-assembler/pull/3785.

jlebon commented 1 month ago

Let's see if https://github.com/coreos/coreos-assembler/pull/3792 helps (then here, we'd export TAIL_SLEEP=3 or something).

jlebon commented 1 month ago

/retest

jlebon commented 1 month ago

Ahh and indeed now we clearly see the error for all of them which means we were missing output. Yuck OK, so we need to fix the tail handling. Though I'll see if I can bang on the virtio-serial bug again.

travier commented 1 month ago

Ahh and indeed now we clearly see the error for all of them which means we were missing output. Yuck OK, so we need to fix the tail handling. Though I'll see if I can bang on the virtio-serial bug again.

Thanks! I indeed had a missing change in those commits.

travier commented 1 month ago

Hum, the workaround is not that ugly and only impacts CI here so maybe we should merge it for now until we've fixed this COSA.

travier commented 1 month ago

CI fixes in https://github.com/openshift/release/pull/51750

jlebon commented 1 month ago

and only impacts CI here

So far. :) It's racy so I don't see why it couldn't happen in the prod pipeline. I'd hate for someone to be debugging a failure there and working with incomplete output thinking the error is happening somewhere other than it really is.

I guess though a major hack is to just add a sleep 1 at least on the cosa side.

travier commented 1 month ago

Failing on missing the teamd package.

travier commented 1 month ago

https://gitlab.com/redhat/centos-stream/rpms/NetworkManager/-/commit/2d2f70f74f4950b2dc6eef7e9541f32e6da0544f

travier commented 1 month ago

https://github.com/coreos/fedora-coreos-tracker/issues/1727

travier commented 1 month ago

/retest

travier commented 1 month ago

kdump not working:

systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:01:37 UTC; 207ms ago
   Duration: 79ms
    Process: 2200 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
   Main PID: 2200 (code=exited, status=1/FAILURE)
        CPU: 27ms

May 14 12:01:36 qemu0 systemd[1]: Started kola-runext.service.
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + . /var/opt/kola/extdata/commonlib.sh
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: ++ IFS=' '
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: ++ read -r -a cmdline
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + case "${AUTOPKGTEST_REBOOT_MARK:-}" in
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + is_service_active kdump.service
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + local service=kdump.service
May 14 12:01:36 qemu0 kola-runext-test.sh[2200]: + for _x in {0..60}
May 14 12:01:36 qemu0 kola-runext-test.sh[2201]: ++ systemctl is-active kdump.service
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + '[' inactive '!=' activating ']'
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + break
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + systemctl is-active kdump.service
May 14 12:01:37 qemu0 kola-runext-test.sh[2202]: inactive
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + fatal 'kdump.service failed to start'
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + echo 'kdump.service failed to start'
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: kdump.service failed to start
May 14 12:01:37 qemu0 kola-runext-test.sh[2200]: + exit 1
May 14 12:01:37 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:01:37 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.shared.kdump.crash (101.06s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-05-14T12:01:37Z cli: Unit kola-runext.service exited with code 1
        harness.go:1263: kolet failed: : kolet run-test-unit failed: Process exited with status 1

teamd test not working (expected as we removed team support):

systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:02:35 UTC; 195ms ago
   Duration: 66ms
    Process: 2164 ExecStart=/usr/local/bin/kola-runext-test.sh (code=exited, status=1/FAILURE)
   Main PID: 2164 (code=exited, status=1/FAILURE)
        CPU: 20ms

May 14 12:02:35 qemu0 systemd[1]: Started kola-runext.service.
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: + . /var/opt/kola/extdata/commonlib.sh
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: ++ IFS=' '
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: ++ read -r -a cmdline
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: + team=team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ get_ipv4_for_nic team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ local nic_name=team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ local ip
May 14 12:02:35 qemu0 kola-runext-test.sh[2167]: +++ ip -j addr show team0
May 14 12:02:35 qemu0 kola-runext-test.sh[2168]: +++ jq -r '.[0].addr_info | map(select(.family == "inet")) | .[0].local'
May 14 12:02:35 qemu0 kola-runext-test.sh[2167]: Device "team0" does not exist.
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ ip=
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ '[' -z '' ']'
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ echo 'Error: can not get ip for team0'
May 14 12:02:35 qemu0 kola-runext-test.sh[2165]: ++ exit 1
May 14 12:02:35 qemu0 kola-runext-test.sh[2164]: + nic_ip='Error: can not get ip for team0'
May 14 12:02:35 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:02:35 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
--- FAIL: ext.config.shared.networking.team-dhcp-via-ignition (59.40s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-05-14T12:02:35Z cli: Unit kola-runext.service exited with code 1
        harness.go:1263: kolet failed: : kolet run-test-unit failed: Process exited with status 1

No more ifcfg files:

=== RUN   non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin
systemctl status kola-runext-63.service:
× kola-runext-63.service
     Loaded: loaded (/etc/systemd/system/kola-runext-63.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:24:42 UTC; 248ms ago
   Duration: 59ms
    Process: 5657 ExecStart=/usr/local/bin/kola-runext-nm-ifcfg-rh-plugin (code=exited, status=1/FAILURE)
   Main PID: 5657 (code=exited, status=1/FAILURE)
        CPU: 21ms

May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ DOCUMENTATION_URL=https://docs.okd.io/latest/welcome/index.html
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ BUG_REPORT_URL=https://access.redhat.com/labs/rhir/
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OPENSHIFT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OSTREE_VERSION=416.10.202405141044-0
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' scos == rhcos ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + is_scos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + source /etc/os-release
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ NAME='CentOS Stream CoreOS'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ ID=scos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ ID_LIKE='rhel fedora'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VERSION=416.10.202405141044-0
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VERSION_ID=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VARIANT=CoreOS
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ VARIANT_ID=coreos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ PLATFORM_ID=platform:el10
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ PRETTY_NAME='CentOS Stream CoreOS 416.10.202405141044-0'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ ANSI_COLOR='0;31'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ CPE_NAME=cpe:/o:centos:centos:10::coreos
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ HOME_URL=https://centos.org/
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ DOCUMENTATION_URL=https://docs.okd.io/latest/welcome/index.html
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ BUG_REPORT_URL=https://access.redhat.com/labs/rhir/
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_BUGZILLA_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT='OpenShift Container Platform'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ REDHAT_SUPPORT_PRODUCT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OPENSHIFT_VERSION=4.16
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ++ OSTREE_VERSION=416.10.202405141044-0
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' scos == scos ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' coreos == coreos ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + '[' 0 == 1 ']'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + fatal 'ifcfg-rh plugin not detected on RHCOS/SCOS'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + echo 'ifcfg-rh plugin not detected on RHCOS/SCOS'
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: ifcfg-rh plugin not detected on RHCOS/SCOS
May 14 12:24:42 qemu0 kola-runext-nm-ifcfg-rh-plugin[5657]: + exit 1
May 14 12:24:42 qemu0 systemd[1]: kola-runext-63.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:24:42 qemu0 systemd[1]: kola-runext-63.service: Failed with result 'exit-code'.

Missing AMD ucode:

=== RUN   non-exclusive-test-bucket-0/ext.config.shared.files.amd-ucode-firmware
systemctl status kola-runext-28.service:
× kola-runext-28.service
     Loaded: loaded (/etc/systemd/system/kola-runext-28.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:28:59 UTC; 208ms ago
   Duration: 29ms
    Process: 13697 ExecStart=/usr/local/bin/kola-runext-amd-ucode-firmware (code=exited, status=1/FAILURE)
   Main PID: 13697 (code=exited, status=1/FAILURE)
        CPU: 10ms

May 14 12:28:59 qemu0 systemd[1]: Started kola-runext-28.service.
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + . /var/opt/kola/extdata-28/commonlib.sh
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: ++ IFS=' '
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: ++ read -r -a cmdline
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + ls '/usr/lib/firmware/amd-ucode/microcode*'
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13698]: ls: cannot access '/usr/lib/firmware/amd-ucode/microcode*': No such file or directory
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + fatal 'no AMD microcode found on host'
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + echo 'no AMD microcode found on host'
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: no AMD microcode found on host
May 14 12:28:59 qemu0 kola-runext-amd-ucode-firmware[13697]: + exit 1
May 14 12:28:59 qemu0 systemd[1]: kola-runext-28.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:28:59 qemu0 systemd[1]: kola-runext-28.service: Failed with result 'exit-code'.

Change in systemd timeout:

=== RUN   non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts
systemctl status kola-runext-79.service:
× kola-runext-79.service
     Loaded: loaded (/etc/systemd/system/kola-runext-79.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:29:10 UTC; 1s ago
   Duration: 114ms
    Process: 14540 ExecStart=/usr/local/bin/kola-runext-default-unit-timeouts (code=exited, status=1/FAILURE)
   Main PID: 14540 (code=exited, status=1/FAILURE)
        CPU: 55ms

May 14 12:29:10 qemu0 systemd[1]: Started kola-runext-79.service.
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + . /var/opt/kola/extdata-79/commonlib.sh
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: ++ IFS=' '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: ++ read -r -a cmdline
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14541]: ++ systemctl show --value --property=DefaultTimeoutStartUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultTimeoutStartUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14542]: ++ systemctl show --value --property=DefaultTimeoutStopUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultTimeoutStopUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14543]: ++ systemctl show --value --property=DefaultTimeoutAbortUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultTimeoutAbortUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14544]: ++ systemctl show --value --property=DefaultDeviceTimeoutUSec
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error='DefaultTimeoutStartUsec=45s '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error+='DefaultTimeoutStopUSec=45s '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error+='DefaultTimeoutAbortUSec=45s '
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + error+=DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + timeout='1min 30s'
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14545]: ++ rpm -q --queryformat '%{VERSION}' systemd
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + systemd_version=255.3
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + required_version=252.4
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + [[ 45s == \1\m\i\n\ \3\0\s ]]
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + fatal DefaultTimeoutStartUsec=45s DefaultTimeoutStopUSec=45s DefaultTimeoutAbortUSec=45s DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + echo DefaultTimeoutStartUsec=45s DefaultTimeoutStopUSec=45s DefaultTimeoutAbortUSec=45s DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: DefaultTimeoutStartUsec=45s DefaultTimeoutStopUSec=45s DefaultTimeoutAbortUSec=45s DefaultDeviceTimeoutUSec=45s
May 14 12:29:10 qemu0 kola-runext-default-unit-timeouts[14540]: + exit 1
May 14 12:29:10 qemu0 systemd[1]: kola-runext-79.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:29:10 qemu0 systemd[1]: kola-runext-79.service: Failed with result 'exit-code'.

RHEL major version test:

=== RUN   non-exclusive-test-bucket-0/ext.config.version.rhel-major-version
systemctl status kola-runext-83.service:
× kola-runext-83.service
     Loaded: loaded (/etc/systemd/system/kola-runext-83.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Tue 2024-05-14 12:32:29 UTC; 350ms ago
   Duration: 4.889s
    Process: 15664 ExecStart=/usr/local/bin/kola-runext-rhel-major-version (code=exited, status=1/FAILURE)
   Main PID: 15664 (code=exited, status=1/FAILURE)
        CPU: 4.202s

May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16372]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=10
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 10 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + for pkg in $(rpm -qa)
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16373]: ++ rpm -q '--queryformat=%{NAME}' libjose-12-1.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + name=libjose
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ libjose == \s\h\i\m\-\x\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ libjose == \s\h\i\m\-\a\a\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16375]: ++ rpm -q '--queryformat=%{RELEASE}' libjose-12-1.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16376]: ++ grep -oP 'el\s*\K\d+'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16377]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=10
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 10 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + for pkg in $(rpm -qa)
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16378]: ++ rpm -q '--queryformat=%{NAME}' shared-mime-info-2.3-4.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + name=shared-mime-info
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ shared-mime-info == \s\h\i\m\-\x\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ shared-mime-info == \s\h\i\m\-\a\a\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16381]: ++ grep -oP 'el\s*\K\d+'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16380]: ++ rpm -q '--queryformat=%{RELEASE}' shared-mime-info-2.3-4.el10.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16382]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=10
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 10 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + for pkg in $(rpm -qa)
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16383]: ++ rpm -q '--queryformat=%{NAME}' conmon-2.1.10-2.1.rhaos4.16.el9.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + name=conmon
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ conmon == \s\h\i\m\-\x\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ conmon == \s\h\i\m\-\a\a\6\4 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16386]: ++ grep -oP 'el\s*\K\d+'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16385]: ++ rpm -q '--queryformat=%{RELEASE}' conmon-2.1.10-2.1.rhaos4.16.el9.x86_64
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[16387]: ++ head -1
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + release=9
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + [[ 9 != \1\0 ]]
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + fatal 'Found packages '\''conmon'\'' with release version '\''9'\'' that does not match the OS current version '\''10'\'''
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + echo 'Found packages '\''conmon'\'' with release version '\''9'\'' that does not match the OS current version '\''10'\'''
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: Found packages 'conmon' with release version '9' that does not match the OS current version '10'
May 14 12:32:29 qemu0 kola-runext-rhel-major-version[15664]: + exit 1
May 14 12:32:29 qemu0 systemd[1]: kola-runext-83.service: Main process exited, code=exited, status=1/FAILURE
May 14 12:32:29 qemu0 systemd[1]: kola-runext-83.service: Failed with result 'exit-code'.
May 14 12:32:29 qemu0 systemd[1]: kola-runext-83.service: Consumed 4.202s CPU time, 1.9M memory peak, 0B memory swap peak.
travier commented 1 month ago 
Installing packages: Packages not found: amd-gpu-firmware, intel-gpu-firmware, nvidia-gpu-firmware
travier commented 1 month ago

https://issues.redhat.com/browse/RHEL-36319

travier commented 1 month ago 
    --- FAIL: non-exclusive-test-bucket-0/ext.config.shared.podman.dns (180.51s)
            harness.go:106: TIMEOUT[3m0s]: ssh: sudo ./kolet run-test-unit --deny-reboots kola-runext-69.service
            harness.go:106: TIMEOUT[3m0s]: ssh: journalctl -t kola-runext-test.sh
    --- FAIL: non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking (180.48s)
            harness.go:106: TIMEOUT[3m0s]: ssh: sudo ./kolet run-test-unit --deny-reboots kola-runext-70.service
            harness.go:106: TIMEOUT[3m0s]: ssh: journalctl -t kola-runext-rootless-pasta-networking
2024-05-23T09:56:53Z kola: Test timed out. Adding as candidate for rerun success: ext.config.shared.podman.rootless-systemd
--- FAIL: ext.config.shared.podman.rootless-systemd (904.79s)
        harness.go:106: TIMEOUT[15m0s]: ssh: sudo ./kolet run-test-unit kola-runext.service
        harness.go:106: TIMEOUT[15m0s]: ssh: journalctl -t kola-runext-rootless-systemd
travier commented 3 weeks ago 
error: Installing packages: Could not depsolve transaction; 1 problem detected:
 Problem: conflicting requests
  - nothing provides containernetworking-plugins >= 0.8.2-3 needed by cri-o-1.28.6-7.rhaos4.15.git70c2e96.el9.x86_64 from rhel-9.4-server-ose-4.17
  - nothing provides containernetworking-plugins >= 0.8.2-3 needed by cri-o-1.29.4-6.rhaos4.16.git0e93ae2.el9.x86_64 from rhel-9.4-server-ose-4.17
failed to execute cmd-fetch: exit status 1

We'll need new cri-o builds to move forward.

openshift-ci[bot] commented 3 weeks ago

@travier: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/scos-9-build-test-qemu ea3bc4a2b9d88fe66ff904cc01c1a3e920fbe0b4 link true /test scos-9-build-test-qemu
ci/prow/rhcos-9-build-test-qemu ea3bc4a2b9d88fe66ff904cc01c1a3e920fbe0b4 link true /test rhcos-9-build-test-qemu
ci/prow/scos-9-build-test-metal ea3bc4a2b9d88fe66ff904cc01c1a3e920fbe0b4 link true /test scos-9-build-test-metal

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).