search

firecracker-microvm / firecracker

Secure and fast microVMs for serverless computing.
http://firecracker-microvm.io
Apache License 2.0
26.3k stars 1.82k forks source link

[Bug] Latest 6.1 guest kernel config results in kernel panic when booting VM #4881

Open kanpov opened 3 weeks ago

kanpov commented 3 weeks ago

Describe the bug

The x86_64 6.1 guest kernel config as it is currently on master after the https://github.com/firecracker-microvm/firecracker/commit/9157a0cf7188706cef3bbe01316171964f569a76 commit, runs seemingly okay with the latest 6.1 kernel (6.1.114), but produces an unusually larger vmlinux (38MB instead of 29MB), and when booting a VM with it the following happens:

[   12.489510] /dev/root: Can't open blockdev
[   12.489784] VFS: Cannot open root device "vda" or unknown-block(0,0): error -6
[   12.490205] Please append a correct "root=" boot option; here are the available partitions:
[   12.490717] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

To Reproduce

  1. Download 6.1.114 Linux kernel source via tarball from kernel.org
  2. Enter the extracted kernel directory
  3. Copy the full contents (much larger after that commit) of https://github.com/firecracker-microvm/firecracker/blob/main/resources/guest_configs/microvm-kernel-ci-x86_64-6.1.config into .config
  4. Run make -j N vmlinux in the dir with N being the number of cores, I personally used 12 as I hit this bug but I doubt this matters
  5. Copy the produced vmlinux anywhere used to start Firecracker VMs
  6. Start a VM configured with: one ext4 block device that is a root device, sync io engine, and no initramfs is used (and this kernel, of course)
  7. Receive the aforementioned error when booting the VM

Expected behaviour

Everything would build correctly and the VM would start.

Environment

Additional context

Even though the output is the same as https://github.com/firecracker-microvm/firecracker/issues/4816, I'm quite sure this isn't the same issue as I'm not using noapic as a kernel boot arg (mine are console=ttyS0 reboot=k panic=1) and adding/removing it doesn't change anything.

HOWEVER! the way I pinned this issue down to that specific commit is that if you take the commit right before that problematic one and open the relevant guest kernel config in it: https://github.com/firecracker-microvm/firecracker/blob/86a2559b26a4b9a05405aeaa58bab0f7261d71bc/resources/guest_configs/microvm-kernel-ci-x86_64-6.1.config

And do the same steps with that config, everything works perfectly and a 29MB working vmlinux is produced.

Checks

kanpov commented 5 days ago

Is there any progress on this?

JackThomson2 commented 2 days ago

Hi @kanpov,

This issue appears to happen because we build from Amazon Linux which has some patches that allow ACPI to be enabled without PCI. To resolve this you can set CONFIG_PCI=y in your config file, or you can build from Amazon Linux as we do.

A link to our docs on booting with acpi on x86-64 can be found here.

Thanks