search

agherzan / meta-raspberrypi

Yocto/OE BSP layer for the Raspberry Pi boards
https://www.yoctoproject.org/
MIT License
520 stars 407 forks source link

[raspberrypi4-64] kvm-image-minimal #1139

Open jwinarske opened 1 year ago

jwinarske commented 1 year ago

Using tip of kirkstone for meta-raspberrypi (this repo) and meta-virtualization (yocto) I am unable to build kvm-image-minimal.

Build Configuration:
BB_VERSION           = "2.0.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "raspberrypi4-64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.0.6"
TUNE_FEATURES        = "aarch64 armv8a crc cortexa72"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "kirkstone:0118853b5bc6402cc8e43be54cd62520f6c19c62"
meta-raspberrypi     = "kirkstone:dacad9302a92b0b7edf8546cdcad1f8ef753e462"
meta-filesystems     
meta-oe              
meta-multimedia      
meta-networking      
meta-python          
meta-perl            = "kirkstone:1f31570d0795da90083d1dbf28127c90908e30ee"
meta-security        = "kirkstone:c79262a30bd385f5dbb009ef8704a1a01644528e"
meta-selinux         = "kirkstone:a401f4b2816a0b41ce8d9351542658c721935bcd"
meta-virtualization  = "kirkstone:9a94fa2ad76990b0eca40837a98aaf4cd83a7248"
meta-flutter-vm      = "kirkstone:f3be076ea6b99a06d1a45b19e4a62e2b16a56f38"
meta-clang           = "kirkstone:b7c0dcc4bc7f0d273f14cc0a70bcad5555a4be04"
meta-flutter-rpi     = "kirkstone:66e644c23ab04c0314f026c1104166b4168c8b61"
meta-vulkan          = "kirkstone:5dcc1871f4053624e48cc036a7a799605cb930c1"
meta-flutter         = "jw/rpi4_virt:01d8e2d471e9686cab85ee2fe2c90472dc8d028b"
***************************************
PV="5.15.34+gitAUTOINC+e1b976ee4f_0086da6acd"
PN="linux-raspberrypi"
***************************************
KERNEL_FEATURES="  cfg/fs/vfat.scc features/kvm/qemu-kvm-enable.scc cfg/virtio.scc"
***************************************
DISTRO_FEATURES="acl alsa argp bluetooth debuginfod ext2 ipv4 ipv6 largefile pcmcia  usbhost wifi xattr nfs zeroconf pci 3g nfc  vfat seccomp largefile opengl  multiarch wayland vulkan alsa wifi bluetooth usbhost opengl vulkan wayland pam acl xattr pam selinux audit security seccomp virtualization kvm systemd pulseaudio gobject-introspection-data ldconfig"
DISTRO_FEATURES_BACKFILL="pulseaudio sysvinit gobject-introspection-data ldconfig"
DISTRO_FEATURES_BACKFILL_CONSIDERED=" sysvinit"
DISTRO_FEATURES_DEFAULT="acl alsa argp bluetooth debuginfod ext2 ipv4 ipv6 largefile pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g nfc x11 vfat seccomp"
DISTRO_FEATURES_FILTER_NATIVE="api-documentation debuginfod opengl wayland"
DISTRO_FEATURES_FILTER_NATIVESDK="api-documentation debuginfod opengl wayland"
DISTRO_FEATURES_NATIVE="x11 ipv6 xattr"
DISTRO_FEATURES_NATIVESDK="x11"
***************************************
IMAGE_FEATURES="debug-tweaks ssh-server-openssh"
IMAGE_FEATURES_CONFLICTS_overlayfs-etc="package-management"
IMAGE_FEATURES_REPLACES_ssh-server-openssh="ssh-server-dropbear"
***************************************
ERROR: kvm-image-minimal-1.0-r0 do_rootfs: Could not invoke dnf. Command '/__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/dnf -v --rpmverbosity=info -y -c /__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/rootfs/etc/dnf/dnf.conf --setopt=reposdir=/__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/rootfs/etc/yum.repos.d --installroot=/__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/rootfs --setopt=logdir=/__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/temp --repofrompath=oe-repo,/__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/oe-rootfs-repo --nogpgcheck install comp-surf-pbr crosvm crosvm-direct crosvm-plugin dart-sdk filament-samples-vk firecracker flutter-auto jailer kernel-module-kvm kernel-module-kvm-amd kernel-module-kvm-intel libvirt libvirt-libvirtd libvirt-virsh packagegroup-core-boot packagegroup-core-ssh-openssh packagegroup-flutter-test-apps qemu run-postinsts sascha-samples tzdata-americas tzdata-core vkcube vkrunner vulkan-tools locale-base-en-us locale-base-en-gb locale-base-es-us' returned 1:
DNF version: 4.11.1
cachedir: /__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/rootfs/var/cache/dnf
Added oe-repo repo from /__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/kvm-image-minimal/1.0-r0/oe-rootfs-repo
User-Agent: falling back to 'libdnf': could not detect OS or basearch
repo: using cache for: oe-repo
oe-repo: using metadata from Mon 23 Jan 2023 06:43:08 PM UTC.
No match for argument: kernel-module-kvm
No match for argument: kernel-module-kvm-amd
No match for argument: kernel-module-kvm-intel
Error: Unable to find a match: kernel-module-kvm kernel-module-kvm-amd kernel-module-kvm-intel

Describe the results you expected: kernel-module-kvm to be built.

Additional details (revisions used, host distro, etc.): CI job: https://github.com/meta-flutter/meta-flutter/blob/jw/rpi4_virt/.github/workflows/kirkstone-rpi4-64-vulkan.yml

jwinarske commented 1 year ago

In tmp/work-shared/raspberrypi4-64/kernel-source/.kernel-meta/cfg/merge_config_build.log I'm seeing:

Merging .kernel-meta/configs//features/kvm/qemu-kvm.cfg
Value of CONFIG_KVM is redefined by fragment .kernel-meta/configs//features/kvm/qemu-kvm.cfg:
Previous value: CONFIG_KVM=y
New value: CONFIG_KVM=m

Value of CONFIG_TUN is redefined by fragment .kernel-meta/configs//features/kvm/qemu-kvm.cfg:
Previous value: CONFIG_TUN=m
New value: CONFIG_TUN=y

Value of CONFIG_MACVLAN is redefined by fragment .kernel-meta/configs//features/kvm/qemu-kvm.cfg:
Previous value: CONFIG_MACVLAN=y
New value: CONFIG_MACVLAN=m

Merging .kernel-meta/configs//cfg/fs/vfat.cfg
Merging .kernel-meta/configs//features/kvm/qemu-kvm.cfg
Merging .kernel-meta/configs//cfg/virtio.cfg
Value of CONFIG_VSOCKETS is redefined by fragment .kernel-meta/configs//cfg/virtio.cfg:
Previous value: CONFIG_VSOCKETS=m
New value: CONFIG_VSOCKETS=y

make[1]: Entering directory '/__w/meta-flutter/rpi4-64-vulkan-kirkstone/raspberrypi4-64/tmp/work/raspberrypi4_64-poky-linux/linux-raspberrypi/1
_5.15.34+gitAUTOINC+e1b976ee4f_0086da6acd-r0/linux-raspberrypi4_64-preempt-rt-build'
  GEN     Makefile
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
./.tmp.config.56xptw7WlG:1619:warning: unexpected data:  
./.tmp.config.56xptw7WlG:1631:warning: symbol value 'm' invalid for KVM
./.tmp.config.56xptw7WlG:1639:warning: unexpected data:  

...

Value requested for CONFIG_KVM not in final .config
Requested value:  CONFIG_KVM=m
Actual value:     # CONFIG_KVM is not set

Value requested for CONFIG_KVM_INTEL not in final .config
Requested value:  CONFIG_KVM_INTEL=m
Actual value:     

Value requested for CONFIG_KVM_AMD not in final .config
Requested value:  CONFIG_KVM_AMD=m
Actual value:
jwinarske commented 1 year ago

In tmp/work/raspberrypi4_64-poky-linux/linux-raspberrypi/1_5.15.34+gitAUTOINC+e1b976ee4f_0086da6acd-r0/linux-raspberrypi4_64-preempt-rt-build/.config I see, which explains why kernel-module-kvm is not valid:

CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
CONFIG_VHOST_IOTLB=m
CONFIG_VHOST=m
CONFIG_VHOST_MENU=y
CONFIG_VHOST_NET=m
CONFIG_VHOST_VSOCK=m
CONFIG_VHOST_CROSS_ENDIAN_LEGACY=y
CONFIG_TUN=y
CONFIG_TAP=m

virtual/kernel bcm2711_defconfig:

CONFIG_KVM=y

meta-virtualization features/kvm/qemu-kvm.cfg augmentation of bcm2711_defconfig:

CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m

Unclear what's causing this error, unless it can't be built as a module. Which could be resolved with another fragment.

Value requested for CONFIG_KVM not in final .config
Requested value:  CONFIG_KVM=m
Actual value:     # CONFIG_KVM is not set
jwinarske commented 1 year ago

I played around with patching a fragment, and I can't get it to patch after features/kvm/qemu-kvm-enable.scc. Regardless of layer order, do_kernel_metadata() seems to always apply kernel cache fragments last.

For preempt-rt the fragment diff looks like:

CONFIG_ARM64_ERRATUM_834220=y
CONFIG_IRQ_BYPASS_MANAGER=y
CONFIG_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_HAVE_KVM_VCPU_RUN_PID_CHANGE=y
CONFIG_KVM_XFER_TO_GUEST_WORK=y
# CONFIG_NVHE_EL2_DEBUG is not set
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_INTERVAL_TREE=y

The diff is different for non preempt-rt.

When I run menuconfig it's after all the other fragments are applied, so really the above diff fragment needs to be applied last.

Any suggestions on applying fragments in a per-determined order?

jwinarske commented 1 year ago

@zeddii

zeddii commented 1 year ago

There's a couple of things at play, we haven't done much testing of meta-virt as a KVM host .. and by that, I mean that i don't know of any testing that has been done.

As such, the KVM enablement fragment isn't correct for arm as "config KVM" is a menuconfig, versus being a tristate in other architectures.

We could fix the fix that invalid symbol error, by making either the KERNEL_FEATURE being set be arch specific, or by making the features/kvm/qemu-kvm-enable.scc have a conditional within the fragment itself.

Both of those take a bit longer to bubble through the various repos.

To the kernel meta-data question, It isn't that the merging applies kernel-cache fragments last, it applies KERNEL_FEATURES last. We've had this on the various lists a few times, and it should be in the manual now, but something that is put into. KERNEL_FEATURE by a kernel recipe/bbappend is something considered to be a "contract" between the kernel recipe requirement and the runtime. So it isn't supposed to be something that is overridden on the SRC_URI.

That being said, you can override it. You can have a KERNEL_FEATURE defined in a layer. Which is just of course a reference to a fragment that is provided by the layer (and that fragment is in the right search path to be found).

I managed to find the link to where this made it into the docs: https://docs.yoctoproject.org/kernel-dev/common.html#adding-recipe-space-kernel-features

So you'd define your fragment above, and then refer to it as a KERNEL_FEATURE, so that i can be applied last.

And finally another idea is that you can (in the short term) do a :remove on the qemu-kvm-enable.scc value in KERNEL_FEATURES, and have something in the SRC_URI defining similar values to what that fragment currently does (except obviously with CONFIG_KVM=y in your case).

Hope that helps.

jwinarske commented 1 year ago

@zeddii Very helpful. The documentation is exactly what I was missing. Thank you!

jwinarske commented 1 year ago

https://github.com/agherzan/meta-raspberrypi/pull/1141

This PR builds kvm as part of kernel. In addition kernel-module-intel, and kernel-module-amd don't apply. So building kvm-image-minimal is not compatible with arm64 targets.