Help required for implementing firecracker-microvm in (host) linux kernel 4.9

[Hardevsinh-sisignals]

I discovered that your official microvm support is limited to kernel versions 4.14 and 5.10. Also, I discovered that issues #19 and #1929 are related to kernel version 4.9, however I was unable to locate the source numbers for 4.9 support and required fixes.

Can somebody recommend which source files to use?

[raduiliescu]

Hi @Hardevsinh-sisignals,

Are you using 4.9 on the host or on the guest side? Can you share the reasons why you won't be able to use a newer kernel versions?

When we initially launched Firecracker we use the 4.9 kernel, but we are not running anymore our CI with this kernel. For the guest you will need to adapt our the 4.14 kernel config suited for your architecture, to 4.9 and build your own kernel. Let me know if you need any help.

Thanks!

[Hardevsinh-sisignals]

Hi , @raduiliescu, Thank you for the prompt reply, As most of the development till now had been on 4.9 kernel , it will be hard to migrate in current state that is the reason we are still using 4.9 kernel. I am free to adopt 4.14 as guest. Can you offer me any pointers on how to run firecracker-microvm with kernel 4.9 as host?

[raduiliescu]

Hi @Hardevsinh-sisignals ,

As long as you have KVM enabled on your host Firecracker might work on 4.9 kernel, just try it :). Still, since we are haven't been testing with 4.9 for about 2 years, you might face issues - either functional or performace ones.

Thanks!

[Hardevsinh-sisignals]

Hi @raduiliescu Thanks for reply. As you suggest i had tried to use firecracker in (host kernel) kernel-4.9 on Amlogic S905D. It supports GICv2 but can not generate vgic in /proc/interrupts using KVM after so many tries.

While running

curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/actions' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "action_type": "InstanceStart" }' Encountering below error, {"fault_message":"Internal error while starting microVM: Failed to validate KVM support: Missing KVM capabilities: Irqchip"}

Any thing missing ? Any help would be greatly appreciable.

[alsrdn]

@Hardevsinh-sisignals Firecracker uses the IRQ chip emulated by KVM which is the vgic in the case of ARM. In your case it seems that it can't use it because probably there was an error when initialising the vgic in KVM. This is the relevant code in KVM: https://elixir.bootlin.com/linux/v4.9.299/source/virt/kvm/arm/vgic/vgic-init.c#L400

Are you seeing any error messages in the kernel log when KVM is initialised? You should be able to grep by kvm [ since all kvm_* print functions pre-pend the string to the output.

[Hardevsinh-sisignals]

@alsrdn , @raduiliescu Thanks for reply, My hardware is supporting GICv2 but i found that GIC-0 is implemented when working with kernel 4.9 and with upgraded version of kernel firecracker worked as expected.

Can you give any suggestion for kernel 4.9?

Can you provide me repo link of firecracker's older release which is working with kernel 4.9?

[raduiliescu]

Hi @Hardevsinh-sisignals ,

In my first replies I haven't realized you are using an Arm platform. We used kernel 4.9 only on X86. For Arm we always had 4.14, and now 5.10.

As @alsrdn said it seems like you have a kvm related issue. Can you search the kvm related errors in the kernel log/dmesg?

Thanks!

[Hardevsinh-sisignals]

hi @alsrdn , @raduiliescu , I have attached picture with output of "cat /proc/interrupts". Can you provide some pointer according? Any help will be greatly appreciated.

[raduiliescu]

Hi @Hardevsinh-sisignals

The irq chip emulation that is used by Firecracker is done by KVM. This is the interrupt setup on the host, which doesn't tell nothing about what KVM is doing.

Is it possible to share the kernel logs? You can get them via dmesg.

Thanks!

[Hardevsinh-sisignals]

Hi @Hardevsinh-sisignals

The irq chip emulation that is used by Firecracker is done by KVM. This is the interrupt setup on the host, which doesn't tell nothing about what KVM is doing.

Is it possible to share the kernel logs? You can get them via dmesg.

Thanks!

Thanks for the reply @raduiliescu , kernel logs are attached. amlogic.log

[Hardevsinh-sisignals]

you might face issues - either functional or performace ones. @raduiliescu Any suspected major functionality?

[dianpopa]

Hi @raduiliescu Thanks for reply. As you suggest i had tried to use firecracker in (host kernel) kernel-4.9 on Amlogic S905D. It supports GICv2 but can not generate vgic in /proc/interrupts using KVM after so many tries.

While running

curl --unix-socket /tmp/firecracker.socket -i \ -X PUT 'http://localhost/actions' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "action_type": "InstanceStart" }' Encountering below error, {"fault_message":"Internal error while starting microVM: Failed to validate KVM support: Missing KVM capabilities: Irqchip"}

Any thing missing ? Any help would be greatly appreciable.

The error you are seeing comes from Firecracker having its own capabilities check. So, as per the error you receive, it would seem that KVM_CAP_IRQCHIP is needed in order to run Firecracker on ARM. This is actually not right since we do not use KVM_CREATE_IRQCHIP to set up the gic device. We actually use KVM_CREATE_DEVICE which in turn mandates the presence of KVM_CAP_DEVICE_CTRL capability. I posted a PR with the fix. Do you think you could try it out in your setup and let us know the outcome?

Thanks, Diana

[Hardevsinh-sisignals]

The error you are seeing comes from Firecracker having its own capabilities check. So, as per the error you receive, it would seem that KVM_CAP_IRQCHIP is needed in order to run Firecracker on ARM. This is actually not right since we do not use KVM_CREATE_IRQCHIP to set up the gic device. We actually use KVM_CREATE_DEVICE which in turn mandates the presence of KVM_CAP_DEVICE_CTRL capability. I posted a PR with the fix. Do you think you could try it out in your setup and let us know the outcome?

@dianpopa Sure, I will test and thanks for the valuable reply and your efforts.

[Hardevsinh-sisignals]

hi @dianpopa, @raduiliescu , I had tested the suggested change and now got another error related to GIC. Please find error in attached picture. Also, " tools/devtool test " is running. but, it takes too much time. As soon as the testing is finished, I will post the results.

[Hardevsinh-sisignals]

Hi @dianpopa Thanks for your active response, I had attached the report in json format. test_report_json.txt

[dianpopa]

Hi,

Sorry for coming back this late.

From the error you posted (specifically the "Error creating the interrupt controller" seems that indeed your host does not have support for a GIC device (not even the legacy one). I would start by making sure you have configured the ARM gic controller on your host: cat /boot/config-$(uname -r) | grep GIC.

Let me know what you discover.