rawhide: kernel 6.8 not booting on aarch64

dustymabe commented 10 months ago

The kola basic test fails and the console output doesn't really give us a lot of information:

UEFI firmware (version edk2-20231122-14.fc39 built at 00:00:00 on Dec 22 2023)^M
SyncPcrAllocationsAndPcrMask!^M
Tpm2GetCapabilityPcrs - 00000004^M
alg - 4^M
alg - B^M
alg - C^M
alg - D^M
Image type X64 can't be loaded on AARCH64 UEFI system.^M
^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[2J^[[01;01H^[[=3h^[[2J^[[01;01HBdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0)^M
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0)^M
^[[0m^[[37m^[[44m^[[01;01H/^[[01;01H/------------------------------------------------------------------------------\^[[02;01H|                           Boot Option Restoration             
UEFI firmware (version edk2-20231122-14.fc39 built at 00:00:00 on Dec 22 2023)^M
SyncPcrAllocationsAndPcrMask!^M
Tpm2GetCapabilityPcrs - 00000004^M
alg - 4^M
alg - B^M
alg - C^M
alg - D^M
Image type X64 can't be loaded on AARCH64 UEFI system.^M
^[[2J^[[01;01H^[[=3h^[[2J^[[01;01H^[[2J^[[01;01H^[[=3h^[[2J^[[01;01HBdsDxe: loading Boot0003 "Fedora" from HD(2,GPT,AD074112-4225-41B1-B703-6852D2C54447,0x1000,0x3F800)/\EFI\fedora\shimaa64
BdsDxe: starting Boot0003 "Fedora" from HD(2,GPT,AD074112-4225-41B1-B703-6852D2C54447,0x1000,0x3F800)/\EFI\fedora\shimaa64.efi^M
error: ../../grub-core/term/serial.c:217:serial port `com0' isn't found.
^Merror: ../../grub-core/commands/terminal.c:138:terminal `serial' isn't found.
^Merror: ../../grub-core/commands/terminal.c:138:terminal `serial' isn't found.
^M^[[0m^[[30m^[[40m^[[2J^[[01;01H^[[0m^[[37m^[[40m^[[02;32HGRUB version 2.06
^M
^M^[[04;02H/----------------------------------------------------------------------------\^[[05;02H|^[[05;79H|^[[06;02H|^[[06;79H|^[[07;02H|^[[07;79H|^[[08;02H|^[[08;79H|^[[09;02H|^[[09;79H|
^M      Press enter to boot the selected OS, `e' to edit the commands
^M      before booting or `c' for a command-line. ESC to return previous
^M      menu.                                                               ^[[05;80H ^[[0m^[[30m^[[47m^[[05;03H*Fedora CoreOS 40.20240113.91.1 (ostree:0)                                  ^
^M
^MEFI stub: Decompressing Linux Kernel...^M
EFI stub: Using DTB from configuration table^M
EFI stub: Exiting boot services...^M

I tested with kernel-6.8.0-0.rc0.20240109git9f8413c4a66f.1.fc40 and kernel-6.8.0-0.rc0.20240111gitde927f6c0b07.4.fc40 and both fail. Reverting to kernel-6.7.0-68.fc40 gets it booting again.

nullr0ute commented 10 months ago

In fact it looks like it might be a qemu issue, what version of qemu?

https://gitlab.com/qemu-project/qemu/-/issues/1990

So it's a problem with shim that's triggered by a change in tianocore/edk2, there's a work around for edk2 in qemu [1], which will ship as part of qemu 8.2.1 later in Jan, not sure if the Fedora build will need a separate update and I'm not sure which ver is being used in this case.

[1] https://github.com/tianocore/edk2/commit/cee7ba349c0c1ce489001a338a4e28555728b573

dustymabe commented 10 months ago

The version of qemu we're using is just whatever is latest in F39. Here is what is in our COSA image right now (which is where these tests are running):

qemu-kvm-8.1.3-1.fc39
edk2-aarch64-20231122-14.fc39
edk2-ovmf-20231122-14.fc39

Are you saying that the 6.8 kernel won't boot on aarch64 when running qemu from older systems (like Fedora 39 or Fedora 38)?

ausil commented 10 months ago

A rawhide vm boots 6.8.0-0.rc0.20240112git70d201a40823.5.fc40.aarch64 just fine. The VM host has the same versions as the previous comment

qemu-kvm-8.1.3-1.fc39.aarch64
edk2-ovmf-20231122-14.fc39.noarch
edk2-aarch64-20231122-14.fc39.noarch

gursewak1997 commented 10 months ago

Ran a rawhide[aarch64] build with the latest kernel-6.8.0-0.rc2.19.fc40 and it boots up fine and the basic tests are passing.

dustymabe commented 10 months ago

Ran a rawhide[aarch64] build with the latest kernel-6.8.0-0.rc2.19.fc40 and it boots up fine and the basic tests are passing.

Just checking: this was on aarch64?

gursewak1997 commented 10 months ago

Just checking: this was on aarch64?

yes

dustymabe commented 10 months ago

we can close this now that https://github.com/coreos/fedora-coreos-config/pull/2824 and https://github.com/coreos/fedora-coreos-config/pull/2825 merged

dustymabe commented 10 months ago

ok update: kernel-6.8.0-0.rc2.19.fc40 works, but kernel-6.8.0-0.rc2.20240130git861c0981648f.20.fc40.aarch64 doesn't.. Some issue with git snapshot kernels maybe?

nullr0ute commented 10 months ago

qemu 8.2.1 which works around this was released on Jan 29th, I don't see that in Fedora yet. shim 15.8 was tagged last week upstream which also deals with this, but in Fedora it's still at 15.6

I've not seen a specific fix land in the kernel, @gursewak1997 what did you run this on?

runcom commented 10 months ago

but kernel-6.8.0-0.rc2.20240130git861c0981648f.20.fc40.aarch64 doesn't

I can confirm that kernel is in fedora-bootc:eln and that's probably why it doesn't boot (even if you use qemu 8.2.1, still won't boot)

dustymabe commented 10 months ago

@gursewak1997 what did you run this on?

@nullr0ute we're just running this inside a COSA container with qemu (based on F39 with whatever is latest there):

[2024-01-31T22:50:59.480Z]  qemu-img                                aarch64  2:8.1.3-2.fc39                    updates                1.9 M
[2024-01-31T22:50:59.480Z]  qemu-kvm                                aarch64  2:8.1.3-2.fc39                    updates                 11 k

aaradhak commented 9 months ago

Tested the kola basic test with the latest kernel version. It seems to fail with the latest kernel version kernel-6.8.0-0.rc4.20240216git4f5e5092fdbf.39.fc41.aarch64. The test is timing out.

[coreos-assembler]$ kola run basic
__   Skipping kola test pattern "fcos.internet":
  __ https://github.com/coreos/coreos-assembler/pull/1478
__   Skipping kola test pattern "podman.workflow":
  __ https://github.com/coreos/coreos-assembler/pull/1478
__   Skipping kola test pattern "ext.config.files.console-config":
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1654
__  Snoozing kola test pattern "ext.config.var-mount.scsi-id" until Feb 21 2024
  __ https://github.com/coreos/fedora-coreos-tracker/issues/1670
=== RUN   basic
 kola: Test timed out. Adding as candidate for rerun success: basic
--- FAIL: basic (601.56s)
        harness.go:106: TIMEOUT[10m0s]: SSH unsuccessful within allotted timeframe for f7d950ab-12e7-4c26-9a59-c9571acf83f6.
FAIL, output in tmp/kola/qemu-2024-02-19-2148-19650
Error: harness: test suite failed
2024-02-19T21:58:07Z cli: harness: test suite failed

dustymabe commented 9 months ago

Could you test with kernel-6.8.0-0.rc5.41.fc41 ?

aaradhak commented 9 months ago

Yes, the kola basic tests PASS with kernel-6.8.0-0.rc5.41.fc41

--- PASS: basic (49.46s)
    --- PASS: basic/MachineID (0.15s)
    --- PASS: basic/FCOSGrowpart (0.17s)
    --- PASS: basic/PortSSH (0.15s)
    --- PASS: basic/DbusPerms (0.37s)
    --- PASS: basic/ServicesActive (0.16s)
    --- PASS: basic/ReadOnly (0.15s)
    --- PASS: basic/Useradd (0.24s)
PASS, output in tmp/kola/qemu-2024-02-20-1626-18963

dustymabe commented 9 months ago

Yes, the kola basic tests PASS with kernel-6.8.0-0.rc5.41.fc41

ok thanks. The mystery continues. Guess we'll have to wait on the investigation from @jmarrero

dustymabe commented 9 months ago

@aaradhak for now can you bump the pin to kernel-6.8.0-0.rc5.41.fc41 then?

dustymabe commented 9 months ago

OK talked with jforbes and pwhalen earlier and they wanted to run some tests

kernel-6.8.0-0.rc5.41.fc41.aarch64 works fine:

[builder@coreos-aarch64-builder build-arch]$ disk=builds/41.20240227.dev.0/aarch64/fedora-coreos-41.20240227.dev.0-qemu.aarch64.qcow2
virt-cat -a $disk -m /dev/sda3 /loader.1/entries/ostree-1-fedora-coreos.conf
title Fedora CoreOS 41.20240227.dev.0 (ostree:0)
version 1
options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/eb0b6712c976dce53bf45fce9e3672d775143840a2d772b412b26df85e58dd47/0 ignition.platform.id=qemu
linux /ostree/fedora-coreos-eb0b6712c976dce53bf45fce9e3672d775143840a2d772b412b26df85e58dd47/vmlinuz-6.8.0-0.rc5.41.fc41.aarch64
initrd /ostree/fedora-coreos-eb0b6712c976dce53bf45fce9e3672d775143840a2d772b412b26df85e58dd47/initramfs-6.8.0-0.rc5.41.fc41.aarch64.img
abootcfg /ostree/deploy/fedora-coreos/deploy/2b5d2358343c73c903f57317b66097fbe92defe59c77ca72506bd17072229ac1.0/usr/lib/ostree-boot/aboot.cfg
fdtdir /ostree/fedora-coreos-eb0b6712c976dce53bf45fce9e3672d775143840a2d772b412b26df85e58dd47/dtb
aboot /ostree/deploy/fedora-coreos/deploy/2b5d2358343c73c903f57317b66097fbe92defe59c77ca72506bd17072229ac1.0/usr/lib/ostree-boot/aboot.img

kernel-6.8.0-0.rc6.49.fc41.aarch64 works fine:

[builder@coreos-aarch64-builder build-arch]$ disk=builds/41.20240227.dev.1/aarch64/fedora-coreos-41.20240227.dev.1-qemu.aarch64.qcow2
virt-cat -a $disk -m /dev/sda3 /loader.1/entries/ostree-1-fedora-coreos.conf
title Fedora CoreOS 41.20240227.dev.1 (ostree:0)
version 1
options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/410c5c8b04e80f584175af0e43ebe92672cc39e5131fcf4337613de5d9180e6e/0 ignition.platform.id=qemu
linux /ostree/fedora-coreos-410c5c8b04e80f584175af0e43ebe92672cc39e5131fcf4337613de5d9180e6e/vmlinuz-6.8.0-0.rc6.49.fc41.aarch64
initrd /ostree/fedora-coreos-410c5c8b04e80f584175af0e43ebe92672cc39e5131fcf4337613de5d9180e6e/initramfs-6.8.0-0.rc6.49.fc41.aarch64.img
abootcfg /ostree/deploy/fedora-coreos/deploy/cf3669cbda29f0b340bb2c11abe57774b13f6fade38972cdcadc785ceb8e2f61.0/usr/lib/ostree-boot/aboot.cfg
fdtdir /ostree/fedora-coreos-410c5c8b04e80f584175af0e43ebe92672cc39e5131fcf4337613de5d9180e6e/dtb
aboot /ostree/deploy/fedora-coreos/deploy/cf3669cbda29f0b340bb2c11abe57774b13f6fade38972cdcadc785ceb8e2f61.0/usr/lib/ostree-boot/aboot.img

kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.fc41.aarch64 does not work:

[builder@coreos-aarch64-builder build-arch]$ disk=builds/41.20240227.dev.2/aarch64/fedora-coreos-41.20240227.dev.2-qemu.aarch64.qcow2
[builder@coreos-aarch64-builder build-arch]$ virt-cat -a $disk -m /dev/sda3 /loader.1/entries/ostree-1-fedora-coreos.conf
title Fedora CoreOS 41.20240227.dev.2 (ostree:0)
version 1
options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/8ed90637a75925a6764840747284f82d927d68bf9bed8a744a02d12c1aedc6b3/0 ignition.platform.id=qemu
linux /ostree/fedora-coreos-8ed90637a75925a6764840747284f82d927d68bf9bed8a744a02d12c1aedc6b3/vmlinuz-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.fc41.aarch64
initrd /ostree/fedora-coreos-8ed90637a75925a6764840747284f82d927d68bf9bed8a744a02d12c1aedc6b3/initramfs-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.fc41.aarch64.img
abootcfg /ostree/deploy/fedora-coreos/deploy/814afe4daa214634f41f3e22036378f584b5d5e9fb42a9bd1dace2a63d88cd4f.0/usr/lib/ostree-boot/aboot.cfg
fdtdir /ostree/fedora-coreos-8ed90637a75925a6764840747284f82d927d68bf9bed8a744a02d12c1aedc6b3/dtb
aboot /ostree/deploy/fedora-coreos/deploy/814afe4daa214634f41f3e22036378f584b5d5e9fb42a9bd1dace2a63d88cd4f.0/usr/lib/ostree-boot/aboot.img

A new kernel jforbes built with a few fewer characters in the version kernel-6.8.0-0.rc6.20240227git45ec2f5f6.50.fc41.aarch64 does not work.

[builder@coreos-aarch64-builder build-arch]$ disk=builds/41.20240227.dev.3/aarch64/fedora-coreos-41.20240227.dev.3-qemu.aarch64.qcow2
virt-cat -a $disk -m /dev/sda3 /loader.1/entries/ostree-1-fedora-coreos.conf
title Fedora CoreOS 41.20240227.dev.3 (ostree:0)
version 1
options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/7789da7aea5907af1fdcbe062c55214e2ed479e5d6fc0f9354d06d3c0b4d8c95/0 ignition.platform.id=qemu
linux /ostree/fedora-coreos-7789da7aea5907af1fdcbe062c55214e2ed479e5d6fc0f9354d06d3c0b4d8c95/vmlinuz-6.8.0-0.rc6.20240227git45ec2f5f6.50.fc41.aarch64
initrd /ostree/fedora-coreos-7789da7aea5907af1fdcbe062c55214e2ed479e5d6fc0f9354d06d3c0b4d8c95/initramfs-6.8.0-0.rc6.20240227git45ec2f5f6.50.fc41.aarch64.img
abootcfg /ostree/deploy/fedora-coreos/deploy/353f2c68d460a6b0254db8639cac703cf6e94307cc173dbbbac88ef37473268e.0/usr/lib/ostree-boot/aboot.cfg
fdtdir /ostree/fedora-coreos-7789da7aea5907af1fdcbe062c55214e2ed479e5d6fc0f9354d06d3c0b4d8c95/dtb
aboot /ostree/deploy/fedora-coreos/deploy/353f2c68d460a6b0254db8639cac703cf6e94307cc173dbbbac88ef37473268e.0/usr/lib/ostree-boot/aboot.img

Though one thing of interest here.. If I set the sysroot.bootprefix: true OSTree config (which basically prepends the /boot string onto the front of the BLS config entires) along with the known to be working kernel-6.8.0-0.rc5.41.fc41.aarch64 it now fails too:

[builder@coreos-aarch64-builder build-arch]$ disk=builds/41.20240227.dev.4/aarch64/fedora-coreos-41.20240227.dev.4-qemu.aarch64.qcow2
virt-cat -a $disk -m /dev/sda3 /loader.1/entries/ostree-1-fedora-coreos.conf
title Fedora CoreOS 41.20240227.dev.4 (ostree:0)
version 1
options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/55afb49fc1a8bd3c69c5c65f075652da9092efcb83cd9fa63307583c330e25c9/0 ignition.platform.id=qemu
linux /boot/ostree/fedora-coreos-55afb49fc1a8bd3c69c5c65f075652da9092efcb83cd9fa63307583c330e25c9/vmlinuz-6.8.0-0.rc5.41.fc41.aarch64
initrd /boot/ostree/fedora-coreos-55afb49fc1a8bd3c69c5c65f075652da9092efcb83cd9fa63307583c330e25c9/initramfs-6.8.0-0.rc5.41.fc41.aarch64.img
abootcfg /ostree/deploy/fedora-coreos/deploy/fc17e5377840f490d257c830ef4682f333991be4cc641910762b156a31f4bc51.0/usr/lib/ostree-boot/aboot.cfg
fdtdir /boot/ostree/fedora-coreos-55afb49fc1a8bd3c69c5c65f075652da9092efcb83cd9fa63307583c330e25c9/dtb
aboot /ostree/deploy/fedora-coreos/deploy/fc17e5377840f490d257c830ef4682f333991be4cc641910762b156a31f4bc51.0/usr/lib/ostree-boot/aboot.img

So there is definitely something to the theory about string length being a factor. For the linux lines in the BLS from the above data we have:

length 128 -> working
length 128 -> working
length 152 -> not working
length 149 -> not working
length 133 -> not working

Keep in mind that 1. and 5. are actually the same kernel but the BLS entry just has /boot/ prepended on it.

cgwalters commented 9 months ago

Ouch. I guess ostree's use of sha256 here is somewhat excessive. It's tricky to fix though because we need to handle old clients, rollbacks etc.

dustymabe commented 9 months ago

Ouch. I guess ostree's use of sha256 here is somewhat excessive. It's tricky to fix though because we need to handle old clients, rollbacks etc.

Do you think the problem is within OSTree itself? Like you mention we could use less verbosity in our directory/filenaming structure to workaround the problem, but that doesn't mean there isn't a problem. i.e. should there be a fix in GRUB (I'm guessing) for this? Or maybe there is a defined character limit somewhere and we're non-compliant.

dustymabe commented 9 months ago

digging down in on this more. This entry in the bls config does not work:

options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/b034291ee52bfbf7209ab953b714b4cf39ac1f4d2b9ee273dd496607720ef62b/0 ignition.platform.id=qemu root=UUID=7626bf7f-e22c-44c5-820f-66458b83c3b4 rw rootflags=prjquota boot=UUID=53e559d6-57c5-4df8-8776-9cfcc113b8ce
linux /ostree/fedora-coreos-b034291ee52bfbf7209ab953b714b4cf39ac1f4d2b9ee273dd496607720ef62b/vmlinuz0123456789012345678901234567890

While the one less character entry of this does work:

options rw $ignition_firstboot ostree=/ostree/boot.1/fedora-coreos/b034291ee52bfbf7209ab953b714b4cf39ac1f4d2b9ee273dd496607720ef62b/0 ignition.platform.id=qemu root=UUID=7626bf7f-e22c-44c5-820f-66458b83c3b4 rw rootflags=prjquota boot=UUID=53e559d6-57c5-4df8-8776-9cfcc113b8ce
linux /ostree/fedora-coreos-b034291ee52bfbf7209ab953b714b4cf39ac1f4d2b9ee273dd496607720ef62b/vmlinuz012345678901234567890123456789

i.e. 130 characters on the BLS config entry line seems to be the breaking point (130 works 131 doesn't).

nullr0ute commented 9 months ago

i.e. 130 characters on the BLS config entry line seems to be the breaking point (130 works 131 doesn't).

Looking at the kernel cmdline option by default it's limited to 512 in include/uapi/asm-generic/setup.h but then both x86_64/aarch64 bump that to 2048 so from the kernel side it shouldn't be a problem.

martinezjavier commented 9 months ago

Can you try booting doing set blscfg=debug in the grub prompt ?

martinezjavier commented 9 months ago

I think that found it, there's a limit indeed in a grub_file_getline() grub helper that is used by the blscfg module to parse the file. I'll type a patch to increase that limit and do a scratch build for you to test.

martinezjavier commented 9 months ago

@dustymabe @cgwalters @nullr0ute I've launched a grub2 scratch with a patch for you to test: https://koji.fedoraproject.org/koji/taskinfo?taskID=114190880

dustymabe commented 9 months ago

@martinezjavier after doing a local test I don't believe that scratch build fixed the problem.

martinezjavier commented 9 months ago

@lsandov1 can you please take a look to this issue? I couldn't find any place in the blscfg module that could limit the number of bytes read from the BLS and (wrongly) thought the limitation was in grub_file_getline().

But it would be better if you dig on this since I no longer have a setup to do grub development and testing. Thanks!

nullr0ute commented 9 months ago

Thanks @martinezjavier for taking a look for me! Hopefully we're narrowing things down.

lsandov1 commented 9 months ago

@lsandov1 can you please take a look to this issue? I couldn't find any place in the blscfg module that could limit the number of bytes read from the BLS and (wrongly) thought the limitation was in grub_file_getline().

But it would be better if you dig on this since I no longer have a setup to do grub development and testing. Thanks!

@martinezjavier sure. Thanks for the initial debugging.

dustymabe commented 8 months ago

@lsandov1 any updates on this? Is there a better place to follow progress?

dustymabe commented 8 months ago

@lsandov1 I see a commit linked here.. were you able to reproduce? Is there a scratch build we should use to test?

dustymabe commented 8 months ago

ok I was working with @lsandov1 to try to make a reproducer he could use to iterate on test/dev. I used the Fedora Cloud aarch64 image from F40 beta and wrote the following script.

It creates a new GRUB entry with a failure case (kernel file name with one too many characters in it) and one for a success (the user can use the arrows during boot to select this entry).

#!/bin/bash
set -eux -o pipefail
SSHKEY=/path/to/sshkey.pub
GOLDIMG=./Fedora-Cloud-Base-Generic.aarch64-40-1.10.qcow2
if [ ! -f $GOLDIMG ]; then
    curl -LO https://kojipkgs.fedoraproject.org/compose/40/Fedora-40-20240320.0/compose/Cloud/aarch64/images/Fedora-Cloud-Base-Generic.aarch64-40-1.10.qcow2
    curl -LO https://kojipkgs.fedoraproject.org/compose/40/Fedora-40-20240320.0/compose/Cloud/aarch64/images/Fedora-Cloud-images-40_Beta-1.10-aarch64-CHECKSUM
    sha256sum --check Fedora-Cloud-images-40_Beta-1.10-aarch64-CHECKSUM --ignore-missing
fi
IMG="${GOLDIMG}.tmp.qcow2"

cp --reflink=auto $GOLDIMG $IMG
guestfish -a $IMG <<'EOF'
run
mount /dev/sda2 /
glob cp /vmlinuz* /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-vmlinuz
glob cp /vmlinuz* /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-vmlinuz
upload -<<END /loader/entries/1000000000000000000000000000000001.conf
title Fedora GRUB Test (FAIL)
version 1
linux /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-vmlinuz
initrd /initramfs-6.8.0-0.rc6.49.fc40.aarch64.img
options no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 root=UUID=1f75769a-2da3-4586-aed2-4c56b8c9ef0f rootflags=subvol=root
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
END
upload -<<END /loader/entries/1000000000000000000000000000000000.conf
title Fedora GRUB Test (SUCCESS)
version 1
linux /xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-vmlinuz
initrd /initramfs-6.8.0-0.rc6.49.fc40.aarch64.img
options no_timer_check net.ifnames=0 console=tty1 console=ttyS0,115200n8 root=UUID=1f75769a-2da3-4586-aed2-4c56b8c9ef0f rootflags=subvol=root
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
END
write-append /grub2/grub.cfg "set timeout=5\nserial --speed=115200\nterminal_input serial console\nterminal_output serial console\n"
EOF
virt-customize -a $IMG --root-password 'password:grubchars'

# - Use tpm=none because otherwise the `--cloud-init` won't work because the cloud-init ISO
#   will get removed after UEFI fallback system reset. This won't be needed in the future when 
#   the following commit is in most versions of virt-manager/virt-install:
#   https://github.com/virt-manager/virt-manager/commit/ec434948a8384541c56bfa04e4985f4fc709bc76
# - Use --controller type=scsi,model=virtio-scsi to workaround this virt-manager cloudinit bug
#   on aarch64: https://github.com/virt-manager/virt-manager/issues/445
virt-install --import --name tester --arch aarch64 \
    --ram  4096 --vcpus 2  --accelerate --autoconsole text \
    --os-variant=fedora-unknown --boot uefi --network bridge=virbr0 \
    --controller type=scsi,model=virtio-scsi \
    --disk "${IMG}" --tpm=none --cloud-init=ssh-key="${SSHKEY}"

We did confirm this ONLY happens on aarch64. UEFI+x86_64 does not have the problem.

marmijo commented 8 months ago

I tested kernel-6.9.0-0.rc1.17.fc41 on rawhide aarch64 and it built, ran, and passed the kola basic tests without issue. I opened https://github.com/coreos/fedora-coreos-config/pull/2936 to bump the pin to this latest RC kernel.

dustymabe commented 8 months ago

I ran a bisect on this with the builds from the rawhide stream and found:

BISECT TEST RESULTS:
Last known good build: 40.20240127.91.0
First known bad build: 40.20240201.91.0

If we look at the package changes in that jump:

'40.20240201.91.0'
ostree diff commit from: compose:fedora/aarch64/coreos/rawhide^^^^^^^^^^^^^^^^^^^^^^^^ (021c6a3278cb90a6f94d5eab2357202bfdd0d3843e8e6c88f792f1e744b4443a)
ostree diff commit to:   compose:fedora/aarch64/coreos/rawhide^^^^^^^^^^^^^^^^^^^^^^^ (04562274a5c4a028c0b1f94b39a4ba21aa2c79b4a0b17d45dccb6a183e2a0148)
Upgraded:
  NetworkManager 1:1.45.9-1.fc40 -> 1:1.45.9-1.fc40.2
  NetworkManager-cloud-setup 1:1.45.9-1.fc40 -> 1:1.45.9-1.fc40.2
  NetworkManager-libnm 1:1.45.9-1.fc40 -> 1:1.45.9-1.fc40.2
  NetworkManager-team 1:1.45.9-1.fc40 -> 1:1.45.9-1.fc40.2
  NetworkManager-tui 1:1.45.9-1.fc40 -> 1:1.45.9-1.fc40.2
  WALinuxAgent-udev 2.9.1.1-1.fc40 -> 2.9.1.1-3.fc40
  aardvark-dns 1.9.0-1.fc40 -> 1.10.0-1.fc40
  adcli 0.9.2-3.fc39 -> 0.9.2-6.fc40
  afterburn 5.5.1-1.fc40 -> 5.5.1-2.fc40
  afterburn-dracut 5.5.1-1.fc40 -> 5.5.1-2.fc40
  alternatives 1.26-1.fc40 -> 1.26-3.fc40
  attr 2.5.2-1.fc40 -> 2.5.2-3.fc40
  authselect 1.5.0-1.fc40 -> 1.5.0-3.fc40
  authselect-libs 1.5.0-1.fc40 -> 1.5.0-3.fc40
  avahi-libs 0.8-24.fc39 -> 0.8-26.fc40
  basesystem 11-18.fc39 -> 11-20.fc40
  bash 5.2.26-1.fc40 -> 5.2.26-2.fc40
  bash-completion 1:2.11-12.fc39 -> 1:2.11-14.fc40
  bind-libs 32:9.18.21-2.fc40 -> 32:9.18.21-4.fc40
  bind-license 32:9.18.21-2.fc40 -> 32:9.18.21-4.fc40
  bind-utils 32:9.18.21-2.fc40 -> 32:9.18.21-4.fc40
  bootupd 0.2.17-2.fc40 -> 0.2.17-3.fc40
  bsdtar 3.7.2-1.fc40 -> 3.7.2-3.fc40
  bubblewrap 0.8.0-1.fc40 -> 0.8.0-3.fc40
  bzip2 1.0.8-16.fc39 -> 1.0.8-18.fc40
  bzip2-libs 1.0.8-16.fc39 -> 1.0.8-18.fc40
  c-ares 1.25.0-1.fc40 -> 1.25.0-3.fc40
  ca-certificates 2023.2.62_v7.0.401-4.fc40 -> 2023.2.62_v7.0.401-6.fc40
  catatonit 0.1.7-20.fc40 -> 0.1.7-22.fc40
  chrony 4.5-1.fc40 -> 4.5-3.fc40
  cifs-utils 7.0-2.fc39 -> 7.0-4.fc40
  clevis 19-4.fc39 -> 19-6.fc40
  clevis-dracut 19-4.fc39 -> 19-6.fc40
  clevis-luks 19-4.fc39 -> 19-6.fc40
  clevis-systemd 19-4.fc39 -> 19-6.fc40
  cloud-utils-growpart 0.33-4.fc40 -> 0.33-7.fc40
  composefs 1.0.2-2.fc40 -> 1.0.3-1.fc40
  composefs-libs 1.0.2-2.fc40 -> 1.0.3-1.fc40
  conmon 2:2.1.8-2.fc40 -> 2:2.1.8-4.fc40
  console-login-helper-messages 0.21.3-6.fc39 -> 0.21.3-8.fc40
  console-login-helper-messages-issuegen 0.21.3-6.fc39 -> 0.21.3-8.fc40
  console-login-helper-messages-motdgen 0.21.3-6.fc39 -> 0.21.3-8.fc40
  console-login-helper-messages-profile 0.21.3-6.fc39 -> 0.21.3-8.fc40
  containerd 1.6.23-2.fc40 -> 1.6.23-4.fc40
  containernetworking-plugins 1.3.0-3.fc39 -> 1.4.0-3.fc40
  containers-common 5:0.57.1-1.fc40 -> 5:0.57.1-7.fc40
  containers-common-extra 5:0.57.1-1.fc40 -> 5:0.57.1-7.fc40
  coreos-installer 0.20.0-2.fc40 -> 0.20.0-3.fc40
  coreos-installer-bootinfra 0.20.0-2.fc40 -> 0.20.0-3.fc40
  coreutils 9.4-3.fc40 -> 9.4-6.fc40
  coreutils-common 9.4-3.fc40 -> 9.4-6.fc40
  cracklib 2.9.11-2.fc39 -> 2.9.11-5.fc40
  criu 3.19-2.fc40 -> 3.19-4.fc40
  criu-libs 3.19-2.fc40 -> 3.19-4.fc40
  crypto-policies 20231204-1.git1e3a2e4.fc40 -> 20231204-3.git1e3a2e4.fc40
  cyrus-sasl-gssapi 2.1.28-17.fc40 -> 2.1.28-19.fc40
  cyrus-sasl-lib 2.1.28-17.fc40 -> 2.1.28-19.fc40
  dbus 1:1.14.10-1.fc40 -> 1:1.14.10-3.fc40
  dbus-broker 35-2.fc40 -> 35-4.fc40
  dbus-common 1:1.14.10-1.fc40 -> 1:1.14.10-3.fc40
  dbus-libs 1:1.14.10-1.fc40 -> 1:1.14.10-3.fc40
  device-mapper-multipath 0.9.7-5.fc40 -> 0.9.7-7.fc40
  device-mapper-multipath-libs 0.9.7-5.fc40 -> 0.9.7-7.fc40
  device-mapper-persistent-data 1.0.9-1.fc40 -> 1.0.9-3.fc40
  diffutils 3.10-3.fc39 -> 3.10-5.fc40
  dnsmasq 2.89-7.fc40 -> 2.89-9.fc40
  dosfstools 4.2-8.fc40 -> 4.2-10.fc40
  dracut 059-18.fc40 -> 059-20.fc40
  dracut-network 059-18.fc40 -> 059-20.fc40
  dracut-squash 059-18.fc40 -> 059-20.fc40
  duktape 2.7.0-5.fc39 -> 2.7.0-7.fc40
  e2fsprogs 1.47.0-2.fc39 -> 1.47.0-4.fc40
  e2fsprogs-libs 1.47.0-2.fc39 -> 1.47.0-4.fc40
  efi-filesystem 5-9.fc39 -> 5-11.fc40
  efibootmgr 18-4.fc39 -> 18-6.fc40
  efivar-libs 38-8.fc39 -> 38-10.fc40
  elfutils-default-yama-scope 0.190-4.fc40 -> 0.190-6.fc40
  elfutils-libelf 0.190-4.fc40 -> 0.190-6.fc40
  elfutils-libs 0.190-4.fc40 -> 0.190-6.fc40
  ethtool 2:6.6-1.fc40 -> 2:6.7-1.fc40
  expat 2.5.0-3.fc39 -> 2.5.0-5.fc40
  file 5.45-1.fc40 -> 5.45-3.fc40
  file-libs 5.45-1.fc40 -> 5.45-3.fc40
  filesystem 3.18-6.fc39 -> 3.18-8.fc40
  findutils 1:4.9.0-6.fc40 -> 1:4.9.0-8.fc40
  flatpak-session-helper 1.15.6-1.fc40 -> 1.15.6-3.fc40
  fmt 10.2.1-1.fc40 -> 10.2.1-3.fc40
  fstrm 0.6.1-8.fc39 -> 0.6.1-10.fc40
  fuse 2.9.9-17.fc39 -> 2.9.9-21.fc40
  fuse-common 3.16.2-1.fc40 -> 3.16.2-3.fc40
  fuse-libs 2.9.9-17.fc39 -> 2.9.9-21.fc40
  fuse-sshfs 3.7.3-5.fc40 -> 3.7.3-7.fc40
  fuse3 3.16.2-1.fc40 -> 3.16.2-3.fc40
  fuse3-libs 3.16.2-1.fc40 -> 3.16.2-3.fc40
  gawk 5.3.0-1.fc40 -> 5.3.0-3.fc40
  gdbm 1:1.23-4.fc39 -> 1:1.23-6.fc40
  gdbm-libs 1:1.23-4.fc39 -> 1:1.23-6.fc40
  gdisk 1.0.9-6.fc39 -> 1.0.9-8.fc40
  gettext-envsubst 0.22-2.fc39 -> 0.22-4.fc40
  gettext-libs 0.22-2.fc39 -> 0.22-4.fc40
  gettext-runtime 0.22-2.fc39 -> 0.22-4.fc40
  git-core 2.43.0-1.fc40 -> 2.43.0-3.fc40
  glibc 2.38.9000-35.fc40 -> 2.38.9000-37.fc40
  glibc-common 2.38.9000-35.fc40 -> 2.38.9000-37.fc40
  glibc-gconv-extra 2.38.9000-35.fc40 -> 2.38.9000-37.fc40
  glibc-minimal-langpack 2.38.9000-35.fc40 -> 2.38.9000-37.fc40
  gmp 1:6.2.1-5.fc39 -> 1:6.2.1-8.fc40
  gpgme 1.22.0-1.fc40 -> 1.22.0-4.fc40
  grep 3.11-5.fc40 -> 3.11-7.fc40
  grub2-common 1:2.06-110.fc40 -> 1:2.06-118.fc40
  grub2-efi-aa64 1:2.06-110.fc40 -> 1:2.06-118.fc40
  grub2-tools 1:2.06-110.fc40 -> 1:2.06-118.fc40
  grub2-tools-minimal 1:2.06-110.fc40 -> 1:2.06-118.fc40
  gvisor-tap-vsock-gvforwarder 6:0.7.2-1.fc40 -> 6:0.7.2-2.fc40
  gzip 1.12-6.fc39 -> 1.12-8.fc40
  hostname 3.23-10.fc40 -> 3.23-12.fc40
  ignition 2.17.0-1.fc40 -> 2.17.0-4.fc40
  inih 57-3.fc40 -> 58-1.fc40
  ipcalc 1.0.3-2.fc39 -> 1.0.3-9.fc40
  iptables-legacy 1.8.10-5.fc40 -> 1.8.10-7.fc40
  iptables-legacy-libs 1.8.10-5.fc40 -> 1.8.10-7.fc40
  iptables-libs 1.8.10-5.fc40 -> 1.8.10-7.fc40
  iptables-nft 1.8.10-5.fc40 -> 1.8.10-7.fc40
  iptables-services 1.8.10-5.fc40 -> 1.8.10-7.fc40
  iptables-utils 1.8.10-5.fc40 -> 1.8.10-7.fc40
  iputils 20231222-1.fc40 -> 20231222-2.fc40
  irqbalance 2:1.9.2-2.fc39 -> 2:1.9.2-4.fc40
  iscsi-initiator-utils 6.2.1.9-17.gitc26218d.fc39 -> 6.2.1.9-20.gita65a472.fc40
  iscsi-initiator-utils-iscsiuio 6.2.1.9-17.gitc26218d.fc39 -> 6.2.1.9-20.gita65a472.fc40
  isns-utils-libs 0.101-7.fc39 -> 0.101-9.fc40
  jansson 2.13.1-7.fc39 -> 2.13.1-9.fc40
  jemalloc 5.3.0-4.fc39 -> 5.3.0-6.fc40
  jq 1.7.1-1.fc40 -> 1.7.1-4.fc40
  json-c 0.17-1.fc40 -> 0.17-3.fc40
  json-glib 1.8.0-1.fc40 -> 1.8.0-3.fc40
  kbd 2.6.4-1.fc40 -> 2.6.4-3.fc40
  kbd-legacy 2.6.4-1.fc40 -> 2.6.4-3.fc40
  kbd-misc 2.6.4-1.fc40 -> 2.6.4-3.fc40
  kernel 6.7.0-68.fc40 -> 6.8.0-0.rc2.19.fc40
  kernel-core 6.7.0-68.fc40 -> 6.8.0-0.rc2.19.fc40
  kernel-modules 6.7.0-68.fc40 -> 6.8.0-0.rc2.19.fc40
  kernel-modules-core 6.7.0-68.fc40 -> 6.8.0-0.rc2.19.fc40
  keyutils 1.6.3-1.fc40 -> 1.6.3-3.fc40
  keyutils-libs 1.6.3-1.fc40 -> 1.6.3-3.fc40
  kmod 31-3.fc40 -> 31-5.fc40
  kmod-libs 31-3.fc40 -> 31-5.fc40
  kpartx 0.9.7-5.fc40 -> 0.9.7-7.fc40
  krb5-libs 1.21.2-3.fc40 -> 1.21.2-5.fc40
  less 643-1.fc40 -> 643-3.fc40
  libaio 0.3.111-16.fc39 -> 0.3.111-19.fc40
  libarchive 3.7.2-1.fc40 -> 3.7.2-3.fc40
  libargon2 20190702-3.fc39 -> 20190702-5.fc40
  libassuan 2.5.6-2.fc39 -> 2.5.6-4.fc40
  libatomic 14.0.1-0.3.fc40 -> 14.0.1-0.4.fc40
  libattr 2.5.2-1.fc40 -> 2.5.2-3.fc40
  libbasicobjects 0.1.1-54.fc39 -> 0.1.1-56.fc40
  libbpf 2:1.2.0-1.fc40 -> 2:1.2.0-3.fc40
  libbsd 0.11.7-5.fc39 -> 0.11.7-7.fc40
  libcap 2.69-1.fc40 -> 2.69-3.fc40
  libcap-ng 0.8.4-2.fc40 -> 0.8.4-4.fc40
  libcbor 0.10.2-3.fc40 -> 0.10.2-5.fc40
  libcollection 0.7.0-54.fc39 -> 0.7.0-56.fc40
  libcom_err 1.47.0-2.fc39 -> 1.47.0-4.fc40
  libdaemon 0.14-26.fc39 -> 0.14-29.fc40
  libdb 5.3.28-59.fc40 -> 5.3.28-61.fc40
  libdhash 0.5.0-54.fc39 -> 0.5.0-56.fc40
  libeconf 0.5.2-1.fc40 -> 0.5.2-3.fc40
  libedit 3.1-48.20230828cvs.fc40 -> 3.1-50.20230828cvs.fc40
  libevent 2.1.12-9.fc39 -> 2.1.12-11.fc40
  libffi 3.4.4-5.fc40 -> 3.4.4-7.fc40
  libfido2 1.14.0-1.fc40 -> 1.14.0-3.fc40
  libgcc 14.0.1-0.3.fc40 -> 14.0.1-0.4.fc40
  libgcrypt 1.10.3-1.fc40 -> 1.10.3-3.fc40
  libgpg-error 1.47-2.fc39 -> 1.47-4.fc40
  libgudev 238-2.fc39 -> 238-5.fc40
  libgusb 0.4.8-1.fc40 -> 0.4.8-3.fc40
  libibverbs 48.0-2.fc40 -> 48.0-4.fc40
  libidn2 2.3.4-3.fc39 -> 2.3.7-1.fc40
  libini_config 1.3.1-54.fc39 -> 1.3.1-56.fc40
  libjcat 0.2.1-1.fc40 -> 0.2.1-2.fc40
  libkcapi 1.4.0-7.fc39 -> 1.4.0-10.fc40
  libkcapi-hmaccalc 1.4.0-7.fc39 -> 1.4.0-10.fc40
  libksba 1.6.5-1.fc40 -> 1.6.5-3.fc40
  libldb 2.8.0-1.fc40 -> 2.8.0-3.fc40
  libluksmeta 9-16.fc39 -> 9-18.fc40
  libmaxminddb 1.9.1-1.fc40 -> 1.9.1-2.fc40
  libmd 1.1.0-2.fc39 -> 1.1.0-4.fc40
  libmnl 1.0.5-3.fc39 -> 1.0.5-5.fc40
  libmodulemd 2.15.0-5.fc39 -> 2.15.0-8.fc40
  libndp 1.8-6.fc39 -> 1.8-9.fc40
  libnet 1.3-1.fc40 -> 1.3-3.fc40
  libnetfilter_conntrack 1.0.9-2.fc39 -> 1.0.9-5.fc40
  libnfnetlink 1.0.1-24.fc39 -> 1.0.1-27.fc40
  libnftnl 1.2.6-3.fc40 -> 1.2.6-5.fc40
  libnghttp2 1.59.0-1.fc40 -> 1.59.0-2.fc40
  libnl3 3.9.0-1.fc40 -> 3.9.0-3.fc40
  libnl3-cli 3.9.0-1.fc40 -> 3.9.0-3.fc40
  libnvme 1.7.1-1.fc40 -> 1.7.1-3.fc40
  libpath_utils 0.2.1-54.fc39 -> 0.2.1-56.fc40
  libpcap 14:1.10.4-2.fc39 -> 14:1.10.4-4.fc40
  libpkgconf 1.9.5-2.fc39 -> 1.9.5-4.fc40
  libpsl 0.21.5-1.fc40 -> 0.21.5-3.fc40
  libpwquality 1.4.5-6.fc39 -> 1.4.5-9.fc40
  libref_array 0.1.5-54.fc39 -> 0.1.5-56.fc40
  librepo 1.17.0-1.fc40 -> 1.17.0-3.fc40
  libreport-filesystem 2.17.11-5.fc40 -> 2.17.11-6.fc40
  libseccomp 2.5.3-6.fc39 -> 2.5.3-8.fc40
  libselinux 3.6-2.fc40 -> 3.6-4.fc40
  libselinux-utils 3.6-2.fc40 -> 3.6-4.fc40
  libsemanage 3.6-1.fc40 -> 3.6-3.fc40
  libsepol 3.6-1.fc40 -> 3.6-3.fc40
  libslirp 4.7.0-4.fc39 -> 4.7.0-6.fc40
  libsmbclient 2:4.19.4-3.fc40 -> 2:4.19.4-4.fc40
  libsolv 0.7.27-2.fc40 -> 0.7.27-4.fc40
  libss 1.47.0-2.fc39 -> 1.47.0-4.fc40
  libstdc++ 14.0.1-0.3.fc40 -> 14.0.1-0.4.fc40
  libtalloc 2.4.1-1.fc40 -> 2.4.1-3.fc40
  libtasn1 4.19.0-3.fc39 -> 4.19.0-6.fc40
  libtdb 1.4.9-1.fc40 -> 1.4.9-3.fc40
  libteam 1.32-1.fc40 -> 1.32-4.fc40
  libtevent 0.16.0-1.fc40 -> 0.16.0-3.fc40
  libtirpc 1.3.4-1.rc1.fc40 -> 1.3.4-1.rc2.fc40.2
  libunistring 1.1-5.fc40 -> 1.1-7.fc40
  libusb1 1.0.26-4.fc40 -> 1.0.26-6.fc40
  libuser 0.64-5.fc40 -> 0.64-7.fc40
  libutempter 1.2.1-10.fc39 -> 1.2.1-13.fc40
  libuv 1:1.47.0-3.fc40 -> 1:1.47.0-6.fc40
  libverto 0.3.2-6.fc39 -> 0.3.2-8.fc40
  libwbclient 2:4.19.4-3.fc40 -> 2:4.19.4-4.fc40
  libxcrypt 4.4.36-4.fc40 -> 4.4.36-5.fc40
  libxml2 2.12.4-1.fc40 -> 2.12.4-3.fc40
  libxmlb 0.3.15-1.fc40 -> 0.3.15-3.fc40
  libyaml 0.2.5-12.fc39 -> 0.2.5-14.fc40
  libzstd 1.5.5-4.fc39 -> 1.5.5-5.fc40
  linux-atm-libs 2.5.1-38.fc40 -> 2.5.1-39.fc40
  lmdb-libs 0.9.31-2.fc39 -> 0.9.32-1.fc40
  logrotate 3.21.0-4.fc39 -> 3.21.0-6.fc40
  lsof 4.98.0-2.fc40 -> 4.98.0-4.fc40
  lua-libs 5.4.6-3.fc39 -> 5.4.6-5.fc40
  luksmeta 9-16.fc39 -> 9-18.fc40
  lz4-libs 1.9.4-4.fc39 -> 1.9.4-6.fc40
  lzo 2.10-9.fc39 -> 2.10-12.fc40
  mdadm 4.2-6.fc39 -> 4.2-8.fc40
  moby-engine 24.0.5-1.fc40 -> 24.0.5-3.fc40
  mokutil 2:0.6.0-7.fc39 -> 2:0.6.0-9.fc40
  mpfr 4.2.1-1.fc40 -> 4.2.1-3.fc40
  nano 7.2-4.fc39 -> 7.2-6.fc40
  nano-default-editor 7.2-4.fc39 -> 7.2-6.fc40
  ncurses 6.4-9.20240113.fc40 -> 6.4-11.20240113.fc40
  ncurses-base 6.4-9.20240113.fc40 -> 6.4-11.20240113.fc40
  ncurses-libs 6.4-9.20240113.fc40 -> 6.4-11.20240113.fc40
  net-tools 2.0-0.67.20160912git.fc39 -> 2.0-0.69.20160912git.fc40
  netavark 1.10.1-1.fc40 -> 1.10.1-5.fc40
  nettle 3.9.1-2.fc39 -> 3.9.1-6.fc40
  newt 0.52.24-1.fc40 -> 0.52.24-3.fc40
  nftables 1:1.0.9-1.fc40 -> 1:1.0.9-3.fc40
  nmstate 2.2.21-1.fc40 -> 2.2.21-3.fc40
  npth 1.6-14.fc39 -> 1.6-18.fc40
  nss-altfiles 2.23.0-1.fc39 -> 2.23.0-3.fc40
  numactl-libs 2.0.16-3.fc39 -> 2.0.16-5.fc40
  nvme-cli 2.7.1-1.fc40 -> 2.7.1-3.fc40
  oniguruma 6.9.9-1.fc40 -> 6.9.9-3.fc40
  openssh 9.6p1-1.fc40 -> 9.6p1-1.fc40.2
  openssh-clients 9.6p1-1.fc40 -> 9.6p1-1.fc40.2
  openssh-server 9.6p1-1.fc40 -> 9.6p1-1.fc40.2
  openssl 1:3.1.4-1.fc40 -> 1:3.1.4-4.fc40
  openssl-libs 1:3.1.4-1.fc40 -> 1:3.1.4-4.fc40
  os-prober 1.81-4.fc39 -> 1.81-6.fc40
  ostree 2024.1-1.fc40 -> 2024.1-2.fc40
  ostree-libs 2024.1-1.fc40 -> 2024.1-2.fc40
  passim-libs 0.1.5-1.fc40 -> 0.1.5-3.fc40
  passt 0^20231230.gf091893-1.fc40 -> 0^20231230.gf091893-3.fc40
  passt-selinux 0^20231230.gf091893-1.fc40 -> 0^20231230.gf091893-3.fc40
  passwd 0.80-15.fc39 -> 0.80-18.fc40
  pcre2 10.42-2.fc40 -> 10.42-2.fc40.2
  pcre2-syntax 10.42-2.fc40 -> 10.42-2.fc40.2
  pigz 2.8-2.fc40 -> 2.8-4.fc40
  pkgconf 1.9.5-2.fc39 -> 1.9.5-4.fc40
  pkgconf-m4 1.9.5-2.fc39 -> 1.9.5-4.fc40
  pkgconf-pkg-config 1.9.5-2.fc39 -> 1.9.5-4.fc40
  policycoreutils 3.6-1.fc40 -> 3.6-3.fc40
  polkit 124-1.fc40 -> 124-2.fc40
  polkit-libs 124-1.fc40 -> 124-2.fc40
  polkit-pkla-compat 0.1-26.fc39 -> 0.1-28.fc40
  popt 1.19-3.fc39 -> 1.19-6.fc40
  procps-ng 4.0.4-1.fc40 -> 4.0.4-3.fc40
  protobuf-c 1.5.0-1.fc40 -> 1.5.0-3.fc40
  psmisc 23.6-4.fc39 -> 23.6-6.fc40
  publicsuffix-list-dafsa 20240107-1.fc40 -> 20240107-3.fc40
  qemu-user-static-x86 2:8.2.0-2.fc40 -> 2:8.2.0-6.fc40
  readline 8.2-7.fc40 -> 8.2-8.fc40
  rpcbind 1.2.6-4.rc2.fc39.1 -> 1.2.6-4.rc2.fc40.4
  rpm 4.19.1-2.fc40 -> 4.19.1-4.fc40
  rpm-libs 4.19.1-2.fc40 -> 4.19.1-4.fc40
  rpm-plugin-selinux 4.19.1-2.fc40 -> 4.19.1-4.fc40
  rpm-sequoia 1.5.0-2.fc40 -> 1.6.0-2.fc40
  rsync 3.2.7-5.fc40 -> 3.2.7-7.fc40
  runc 2:1.1.9-1.fc40 -> 2:1.1.9-2.fc40
  samba-client-libs 2:4.19.4-3.fc40 -> 2:4.19.4-4.fc40
  samba-common 2:4.19.4-3.fc40 -> 2:4.19.4-4.fc40
  samba-common-libs 2:4.19.4-3.fc40 -> 2:4.19.4-4.fc40
  sed 4.8-14.fc39 -> 4.8-15.fc40
  setup 2.14.5-1.fc40 -> 2.14.5-2.fc40
  sg3_utils 1.46-6.fc39 -> 1.46-7.fc40
  sg3_utils-libs 1.46-6.fc39 -> 1.46-7.fc40
  shadow-utils 2:4.14.0-2.fc40 -> 2:4.14.0-4.fc40
  shadow-utils-subid 2:4.14.0-2.fc40 -> 2:4.14.0-4.fc40
  shared-mime-info 2.3-2.fc40 -> 2.3-3.fc40
  skopeo 1:1.14.1-1.fc40 -> 1:1.14.1-2.fc40
  slang 2.3.3-4.fc40 -> 2.3.3-5.fc40
  slirp4netns 1.2.2-1.fc40 -> 1.2.2-2.fc40
  snappy 1.1.10-2.fc39 -> 1.1.10-4.fc40
  socat 1.7.4.4-3.fc39 -> 1.7.4.4-5.fc40
  spdlog 1.12.0-2.fc39 -> 1.12.0-3.fc40
  sqlite-libs 3.45.0-1.fc40 -> 3.45.0-2.fc40
  squashfs-tools 4.6.1-2.fc39 -> 4.6.1-4.fc40
  ssh-key-dir 0.1.4-5.fc40 -> 0.1.4-6.fc40
  tar 2:1.35-2.fc40 -> 2:1.35-3.fc40
  teamd 1.32-1.fc40 -> 1.32-4.fc40
  toolbox 0.0.99.5-3.fc40 -> 0.0.99.5-4.fc40
  tpm2-tools 5.6-1.fc40 -> 5.6-2.fc40
  tpm2-tss 4.0.1-6.fc40 -> 4.0.1-7.fc40
  tpm2-tss-fapi 4.0.1-6.fc40 -> 4.0.1-7.fc40
  tzdata 2023d-2.fc40 -> 2023d-3.fc40
  userspace-rcu 0.14.0-3.fc39 -> 0.14.0-4.fc40
  vim-data 2:9.1.031-1.fc40 -> 2:9.1.031-2.fc40
  vim-minimal 2:9.1.031-1.fc40 -> 2:9.1.031-2.fc40
  wasmedge-rt 0.13.5-1.fc40 -> 0.13.5-2.fc40
  which 2.21-40.fc39 -> 2.21-41.fc40
  wireguard-tools 1.0.20210914-5.fc39 -> 1.0.20210914-6.fc40
  xfsprogs 6.5.0-1.fc40 -> 6.5.0-2.fc40
  xxhash-libs 0.8.2-1.fc39 -> 0.8.2-2.fc40
  xz 5.4.5-1.fc40 -> 5.4.6-1.fc40
  xz-libs 5.4.5-1.fc40 -> 5.4.6-1.fc40
  yajl 2.1.0-22.fc39 -> 2.1.0-23.fc40
  zchunk-libs 1.4.0-1.fc40 -> 1.4.0-2.fc40
  zincati 0.0.27-1.fc40 -> 0.0.27-2.fc40
  zlib-ng-compat 2.1.6-1.fc40 -> 2.1.6-2.fc40
  zram-generator 1.1.2-8.fc39 -> 1.1.2-9.fc40
  zstd 1.5.5-4.fc39 -> 1.5.5-5.fc40
Removed:
  libicu-73.2-3.fc40.aarch64
Added:
  libicu73-73.2-1.fc40.aarch64

So this was the first successful build after the F40 mass rebuild so every package changed :).

Though I do notice this too:

  kernel 6.7.0-68.fc40 -> 6.8.0-0.rc2.19.fc40

Which I narrowed down in on. There were a few RC kernels for 6.8 in rawhide that weren't part of the rawhide FCOS stream because the failed to pass tests (see original description of this issue). It appears that this problem was introduced in the very first 6.8 kernel build: kernel-6.8.0-0.rc0.20240109git9f8413c4a66f.1.fc40.aarch64.

dustymabe commented 8 months ago

xref: Fedora bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=2272761

dustymabe commented 8 months ago

Looks like the latest 6.9 kernels aren't showing this problem any longer (even rc git snapshot ones): https://bugzilla.redhat.com/show_bug.cgi?id=2272761#c6

dustymabe commented 7 months ago

closing this out since we are no longer carrying a pin for this.

coreos / fedora-coreos-tracker

rawhide: kernel 6.8 not booting on aarch64 #1647