rawhide: 38.20220930.91.0: `files.validate-symlinks` test fails

[gursewak1997]

Looks like the upgrade for linux-firmware resulted in ext.config.files.validate-symlinks failure. Pipeline: link Failure:

Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ -e /usr/lib/firmware/qcom/sc8280xp/LENOVO/21BX.xz ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + found=false
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\m\t\a\b* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\m\t\a\b ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\s\s\l\/* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\s\s\l\/ ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\s\w\i\d\/\s\w\i\d\t\a\g\s\.\d\/\f\e\d\o\r\a\p\r\o\j\e\c\t\.\o\r\g* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\s\w\i\d\/\s\w\i\d\t\a\g\s\.\d\/\f\e\d\o\r\a\p\r\o\j\e\c\t\.\o\r\g ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\x\d\g\/\s\y\s\t\e\m\d\/\u\s\e\r* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\e\t\c\/\x\d\g\/\s\y\s\t\e\m\d\/\u\s\e\r ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\l\i\b\/\.\b\u\i\l\d\-\i\d\/* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\l\i\b\/\.\b\u\i\l\d\-\i\d\/ ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\l\i\b\/\f\i\r\m\w\a\r\e\/\b\r\c\m\/\b\r\c\m\f\m\a\c\4\3\4\5\5\-\s\d\i\o\.\A\W\-\C\M\2\5\6\S\M\.\t\x\t\.\x\z* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\l\i\b\/\f\i\r\m\w\a\r\e\/\b\r\c\m\/\b\r\c\m\f\m\a\c\4\3\4\5\5\-\s\d\i\o\.\A\W\-\C\M\2\5\6\S\M\.\t\x\t\.\x\z ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\l\i\b\/\m\o\d\u\l\e\s\/* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\l\i\b\/\m\o\d\u\l\e\s\/ ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\e\t\c\-\p\k\i\-\e\n\t\i\t\l\e\m\e\n\t* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\e\t\c\-\p\k\i\-\e\n\t\i\t\l\e\m\e\n\t ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\e\d\h\a\t\.\r\e\p\o* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\e\d\h\a\t\.\r\e\p\o ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + for search_element in "${list_broken_symlinks_skip[@]}"
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\h\s\m* ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ /usr/lib/firmware/qcom/LENOVO/21BX.xz == \/\u\s\r\/\s\h\a\r\e\/\r\h\e\l\/\s\e\c\r\e\t\s\/\r\h\s\m ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + [[ false == \f\a\l\s\e ]]
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + fatal 'Error: /usr/lib/firmware/qcom/LENOVO/21BX.xz symlink to /usr/lib/firmware/qcom/sc8280xp/LENOVO/21BX.xz which does not exist'
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + echo 'Error: /usr/lib/firmware/qcom/LENOVO/21BX.xz symlink to /usr/lib/firmware/qcom/sc8280xp/LENOVO/21BX.xz which does not exist'
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: Error: /usr/lib/firmware/qcom/LENOVO/21BX.xz symlink to /usr/lib/firmware/qcom/sc8280xp/LENOVO/21BX.xz which does not exist
Sep 30 17:16:11 qemu0 kola-runext-validate-symlinks[1709]: + exit 1
Sep 30 17:16:11 qemu0 systemd[1]: kola-runext-34.service: Main process exited, code=exited, status=1/FAILURE
Sep 30 17:16:11 qemu0 systemd[1]: kola-runext-34.service: Failed with result 'exit-code'.
Sep 30 17:16:11 qemu0 systemd[1]: kola-runext-34.service: Consumed 1.705s CPU time.

[dustymabe]

This is in F37 now too. This update is the problem one: https://bodhi.fedoraproject.org/updates/FEDORA-2022-96a68748a1

[travier]

Let's just add this folder to the skip must on the test or change the skip list to skip the entire firmware directory

[travier]

After that we can probably file a bug report in bugzilla

[travier]

See https://github.com/coreos/fedora-coreos-config/blob/testing-devel/tests/kola/files/validate-symlinks#L23 for the skip list

[gursewak1997]

So, are we basically just skipping this folder in next-devel and branched or should I do the same for rawhide and drop the package pin to keep the temporary-solution consistent throughout?

Bug: In the latest(linux-firmware-20220913-140) linux-firmware package, the symlink to /usr/lib/firmware/qcom/LENOVO/21BX.xz can't be validated and is hence broken. We tested it after pinning the image to the previous version(linux-firmware-202200815-139) and it works.

[dustymabe]

So, are we basically just skipping this folder in next-devel and branched or should I do the same for rawhide and drop the package pin to keep the temporary-solution consistent throughout?

Drop the package pin in rawhide once we have the test updated. This will probably make it to F36 too, so it will probably affect all streams soon.

[dustymabe]

@gursewak1997 @travier - what are the next steps here?

I think maybe a BZ report as suggested in https://github.com/coreos/fedora-coreos-tracker/issues/1310#issuecomment-1265729988 ?

[travier]

Can you do a regular Fedora installation in a VM with a cloud or container image and verify the symlinks for all ~module~ firmware files? Then we can file a single bug for all of them at the same time.

[gursewak1997]

BZ issue: https://bugzilla.redhat.com/show_bug.cgi?id=2145029