Closed maggie-lou closed 1 month ago
Hi @maggie-lou, I think I know what this is about. I assume your kernel .config
does not include CONFIG_PCI=y
, correct?
If that is a case, please take a look here: https://github.com/firecracker-microvm/firecracker/blob/main/docs/kernel-policy.md#booting-with-acpi-x86_64-only
We are trying to work with the Linux community to patch the kernel, so you don't have to enable CONFIG_PCI
: https://www.spinics.net/lists/linux-acpi/msg125662.html in cases where the hypervisor doesn't use PCI, but in the meantime you should enable it to get it to work.
Could you please try it out and let me know if it works?
@bchalios unfortunately it does already include CONFIG_PCI=y
. We are using the exact kernel config from the firecracker repo (https://github.com/firecracker-microvm/firecracker/blob/main/resources/guest_configs/microvm-kernel-ci-x86_64-5.10.config)
That is weird. Could you provide a link to the source code where you build the kernel from? Btw, according to your logs, you seem to be using guest kernel 5.15 which we do not officially support. We have tested this configuration with 4.14, 5.10 and 6.1 without issues.
Ok, I tried building 5.15 from Linus's tree doing:
git clone --depth=1 --branch v5.15 https://github.com/torvalds/linux
and building it with our config from here: https://github.com/firecracker-microvm/firecracker/blob/main/resources/guest_configs/microvm-kernel-ci-x86_64-5.10.config
I am able to boot a microVM without issues using the default command line parameters. However, using the command line parameters from your guest kernel logs reproduces the issue you are seeing. I'm going to look which parameter causes this.
it looks as if the offending parameter is the noapic
one. If I remove it from the command line the microVM boots correctly. Could you try and apply this and see if the problem goes away for you?
Nice - that fixed it. Thank you!
Describe the bug
After upgrading to v1.8.0, the guest kernel panics when trying to execute a go init script. The panic occurs before the script itself begins executing (i.e. it's not something within the script causing the panic).
To Reproduce
I'm still working on reproducing the panic. In this test repo, when using the same guest kernel image and boot args, the init process hangs but does not panic. The test repo is able to run successfully if ACPI is disabled.
To run the test repo, you can clone https://github.com/maggie-lou/firecracker-repro/tree/init_go; checkout
init_go
; and runmake test
.I've included the full guest kernel logs at the end, if that sparks some recognition and would appreciate any debugging tips.
Within our setup, the panic happens 100% of the time.
Expected behaviour
Before the init script starts executing, we see a panic in the guest kernel logs (Full kernel logs at the end). If we disable ACPI with
acpi=off
in the boot args, the script successfully runs.Environment
We are using this guest kernel config: https://github.com/firecracker-microvm/firecracker/blob/main/resources/guest_configs/microvm-kernel-ci-x86_64-5.10.config
Additional context
Our understanding is that disabling ACPI will be deprecated on x86_64, so we want to make sure we can still initialize VMs.
We are not very familiar with ACPI, and it's very likely the problem is due to misconfiguration on our end. We'd appreciate any guidance if that's the case.
Checks
We have followed these docs and ensured the guest kernel was configured with
CONFIG_ACPI=y CONFIG_PCI=y CONFIG_X86_MPPARSE=n CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=n
We are using the guest kernel config provided in the firecracker repo and have tried both including and not including the following options which were mentioned in the docs, but not included in that sample config.
We narrowed down the source of the panic to ACPI by running a git bisect on the commits in the latest release, and verified that disabling ACPI with
acpi=off
fixes the problems.Full guest kernel logs