service VM kernel?

[liyimeng]

Describe the bug

This is more like a question instead of a bug report. I just wonder what is built into service VM kernel in additional to vanilla one. What about user VM, dose it need to user acrn kernel as well.

I follow startup instruction to build a service VM kernel, it went well. Since I don't wanna use ubuntu 20.04 as my service OS, I copy the built kernel into another OS, alpine and try to boot it. It did not boot, right after decompress the kernel, it stuck at loading kernel. I guess is that I only copy kernel over, but the firmware is missing.

So my questions are:

• will arcn upstream to Linux kernel in the future?
• is it possible to have a pre-built service VM kernel to share with all project, need need to build from scratch every time.
• How to get the linux firmware with the service VM kernel if I have to build one from scratch?

Thanks a lot in advanced!

Platform

ubuntu lts 22.04 acrn v3.0

Codebase

v3.0

Scenario

NA

[gvancuts]

There are a number of ACRN drivers included in the kernel, some of which have been upstreamed but I believe not absolutely all of them. You would also need to use a version of ACRN and the device model (acrn-dm) that can work with the upstream drivers, I'm not on the latest here. @junjiemao1 , can you help us get an answer here?

You should be able to use that acrn-kernel with other distros but you may need some tweaks in the config, and also make sure to install all kernel modules in the rootfs of your distro. Can you share the log of where the kernel stops booting? Were you trying to boot it in a native environment or under ACRN?

The User VM kernel can be the standard kernel, just make sure you've got the VirtIO drivers that you wish to use built in it.

[junjiemao1]

@gvancuts is right. The basic ACRN support (controlled by CONFIG_ACRN_GUEST and CONFIG_ACRN_HSM) is in upstream already. However, there are still some cutting edge features (that are walking through the upstream process) and config tweaks (that are typically set or unset in kernels from distributions). That is why building a kernel for the service VM is recommended.

A built Linux kernel is more than the kernel binary itself; there are a couple of drivers (as kernel modules) and firmware under the /lib/modules/ directory. You may want to run make install and make modules_install with customized INSTALL_PATH and INSTALL_MOD_PATH respectively to collect all the files to be installed and copy them to your target machine.

[liyimeng]

Thanks @gvancuts @junjiemao1 for the nice explanation.

I try to boot the build kernel with QEMU, it stuck like this:

Maybe the kernel only work when running on top of acrn hypervisor?

Would ARCN features be provided as patch set instead of a fork? In my case, I need some other kernel version with custom module and firmware in my service VM. It seem I have to stick to kernel 5.10 as ACRN provided.

[gvancuts]

Maybe the kernel only work when running on top of acrn hypervisor?

How are you starting your QEMU virtual machine? Are you using an EFI virtual bootloader?

As far as I know, you can run the acrn kernel on a bare metal (with the hypervisor), not 100% sure about QEMU but let's first inspect how you start that QEMU VM and how your virtual disk image has been built and updated with the new kernel.

Would ARCN features be provided as patch set instead of a fork? In my case, I need some other kernel version with custom module and firmware in my service VM. It seem I have to stick to kernel 5.10 as ACRN provided.

The problem is that the patchset would be tied to the kernel version (to some extent), so moving to a different kernel version would still require some porting.

@junjiemao1 , do we have a way we can easily identify the set of patches that are needed for ACRN in the kernel?

[liyimeng]

@gvancuts I have tried a couple of ways, always endup with the same result. Here is my last attempt:

1. Create a qemu VM, use it as the development VM and build service VM kernel as instructed from ARCN tutorial.
2. Copy out vmlinuz file that just built, together with initrd file(this file is from my development VM, not from ACRN build script) to host
3. Use virt-manager to boot the kernel, as picture show below:

then it stuck in the screen I show above.

[gvancuts]

I would avoid using the "Direct kernel boot" option and update the virtual disk image (inc. Grub bootloader in it) to boot your new kernel. What OS do you run in the virtual disk image?

[liyimeng]

I have done so -- booting with grub, the same error. Here I just wanna re-produce the problem in a quick way, hence using direct kernel boot. I have try both Alpine and Unbuntu 22.04 as baseline OS.

The problem here is that even vmlinuz is not run, far from loading initramfs, even far from loading rootfs on OS disk. So whatever I have on root disk should not be matter. I guess it is some firmware mssing from vmlinuz file.

[junjiemao1]

To generate the patch series that are for ACRN but not yet upstreamed, you can simply tell git to format all patches of the repo/branch https://github.com/projectacrn/acrn-kernel since commit 184cb3b3a30f179ff12fa9a77df9c61819783a17 (inclusive).

As for running the kernel in QEMU, would you please try adding console=ttyS0 loglevel=7 to the kernel args? Without a console= option the kernel won't print anything to anywhere.

[liyimeng]

@junjiemao1 No, the args make no difference :(

[liyimeng]

by the way, thank for hit for patch set, I run a rough estimation, it seem not a huge number left from upstream.

git format-patch e0c6d7fe7ddc1a19e051e375907052dfab415ed7
0001-config-Add-minimal-config-for-5.10-ACRN-kernel.patch
0002-virt-acrn-Introduce-acrntrace-support.patch
0003-config-add-the-Virtio-blk-driver-to-the-Service-VM-c.patch
0004-config-add-IGC-driver-Intel-Ethernet-I225-LM.patch
0005-config-include-additional-USB-drivers.patch
0006-config-add-AST-driver-ASPEED-GPU.patch
0007-config-Enabled-Intel-I225-NIC-PTM-capability.patch
0008-config-remove-kernel-physical-start-address.patch
0009-config-remove-CONFIG_EXTRA_FIRMWARE-and-CONFIG_EXTRA.patch
0010-virt-acrn-add-acrn-hypervisor-log-back.patch
0011-Kernel-config-Rename-sos-config-to-service_vm-config.patch
0012-config-update-serial-configuration-for-Service-VM.patch
0013-config-Add-kernel_config_user_vm.patch
0014-virt-acrn-Remove-dependency-of-get-platform-info.patch
0015-config-add-TCC-buffer-driver-support.patch
0016-config-Add-Docker-related-configs-to-kernel_config_u.patch
0017-config-Add-the-support-for-UDMABUF.patch
0018-config-Enable-Virtio-GPU-in-guest_vm.patch
0019-ACRN-Kernel-Enable-vsock-kernel-config.patch
0020-config-Enable-UaaG-kernel-console.patch
0021-xfs-Added-XFS-support-in-config-file-s.vm.patch
0022-xfs-Added-XFS-support-in-config-file-u.vm.patch
0023-Configruations-convert-to-minimal-config-files.patch
0024-config-Enable-Service-VM-framebuffer-console.patch
0025-config-Add-CONFIG_FB_EFI-y-to-service-vm-config.patch
0026-config-Add-CONFIG_DEVMEM-y-to-service-VM-config.patch

Would it work when apply such patch set to other kernel version than 5.10?

[liyimeng]

@junjiemao1 according to this https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc.git/commit/?h=char-misc-next&id=666834c47d3b41da550bbcbc709148e5fc14879c

More arcn code is merged in 5.12. why we still stick to 5.10 in acrn repo?

[junjiemao1]

The acrn-kernel repo is now based on 5.15.44 as its Makefile states.

[liyimeng]

I manage to get more printout via adding kernel parameters earlyprintk=vga console=ttyS0 loglevel=7 It stuck at above screen.

[junjiemao1]

From the log it seems the VM is not configured with a COM port 1 properly. You can try removing console=ttyS0 from the command line to make the vga earlyprintk being alive for some more time.

[liyimeng]

Remove console=ttyS0, I will get no outpout at all. My guest is that service VM is being optimised and remove some I/O module, Am I right？

Funny enough though! I add back console=ttyS0 and now it boot! I don't know what I have done different, only append console=ttyS0 in the end, instead of in the middle of params list? Anyway, I am happy now :D

Below is my VM

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>ServiceVM</name>
  <uuid>d72fd81f-bc6f-4bd8-bb15-b6cde0b03808</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/20.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">4194304</memory>
  <currentMemory unit="KiB">4194304</currentMemory>
  <vcpu placement="static">4</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-6.2">hvm</type>
    <kernel>/home/liyi/workspace/liyi/dist/generic/main/boot/vmlinuz-5.15.44-acrns-g1eb64cf45302</kernel>
    <initrd>/home/liyi/workspace/liyi/dist/artifacts/nitrd</initrd>
    <cmdline>earlyprintk=vga loglevel=7</cmdline>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="utc">
    <timer name="tsc" present="yes"/>
    <timer name="kvmclock" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0xe"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0xf"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="16" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:15:76:35"/>
      <source network="default"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="unix">
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <audio id="1" type="none"/>
    <video>
      <model type="vga" vram="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x10" slot="0x01" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
    <rng model="virtio">
      <backend model="random">/dev/urandom</backend>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </rng>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-machine"/>
    <qemu:arg value="kernel-irqchip=split"/>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="Denverton,+invtsc,+lm,+nx,+smep,+smap,+mtrr,+clflushopt,+vmx,+x2apic,+popcnt,-xsave,+sse,+rdrand,-vmx-apicv-vid,+vmx-apicv-xapic,+vmx-apicv-x2apic,+vmx-flexpriority,+tsc-deadline,+pdpe1gb"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="intel-iommu,intremap=on,caching-mode=on,aw-bits=48"/>
  </qemu:commandline>
</domain>

[junjiemao1]

I don't think we apply significant I/O module tailoring in the service VM kernel. Anyway good to know that you have the kernel up!

If there is no further issues, feel free to close this issue.

[liyimeng]

Thanks @junjiemao1 !