PiMox 7.1 with Kernel Update?

[Sysadminfromhell]

Hey there, first: thank you for your amazing work!

I have serval questions and maybe some ideas to make the PiMox more useful / amazing.

1. Kernel Update / Modification needed for PiMox 7 => Proxmox 7 uses KSMTuned, which isnt supported on the default Raspbian OS (Debian 11), you have to modify the Kernel to Support KSM Features, is it possible that you can "ship" the PiMox 7 with the actual kernel latest supported? For Proxmox 7 it would be 5.11 and Proxmox 7.1 it would be 5.13 (afaik)

2. Update PiMox from 7 to 7.1 => When we can count on the Update to happen? The new KSM Features are good for CPU/RAM Management specially for the Pi and ARM64 devices (emulated)

3. Installation of ZRAM instead of SWAP/ZSWAP => I Switched from SWAP to ZRAM and with the KSMTuned its an amazing view to see how beautiful the Pi is running right now. I have 2 VMs (only) and an M2. SATA SSD Storage for the VMs to run which really speedup things here.

I Compiled my own Kernel (5.11.22) with KSM Features activated.

Kind regards,

[the-wondersmith]

So I tried to follow suit and compile a kernel with KSM enabled and promptly b0rked my cluster 🥲

Any chance you could either package up and share the one you compiled or maybe post some directions on how to replicate your set up?

[Sysadminfromhell]

Hi there, unfortunately i do not have my config anymore because I’m experiment with other branches from the original kernel branches (5.10/5.11/5.13/5.14/5.15).

I use the ARM64 Kernel!

I followed the instructions of https://www.raspberrypi.com/documentation/computers/linux_kernel.html

1. Build the Kernel sudo apt install git bc bison flex libssl-dev make git clone --depth=1 --branch <branch> https://github.com/raspberrypi/linux (The branch you want, if you want default (5.10) remove the branch part. cd linux KERNEL=kernel8 make bcm2711_defconfig -j4
2. Configure the kernel sudo apt install libncurses5-dev make menuconfig -j4 Choose KSM Support directly and then under Memory management activate KSM Support ans Free Sample page report or support (i don’t remember)

After all this build the kernel as described in the article. Make it directly on the Pi itself and don’t forget to give the kernel a different name in the menu config, so any upgrade/update wont override it.

[aanon4]

I you are still running your modified kernel, you should be able to get the config you used by doing the following:

modprobe configs zcat /proc/config.gz

[Sysadminfromhell]

I you are still running your modified kernel, you should be able to get the config you used by doing the following:

modprobe configs zcat /proc/config.gz

here u go: Config.txt

[Sysadminfromhell]

New Update: i had to rollback to the 5.10 because of the ZFS-DKMS Module. The ZFS-DKMS from the official DEB only support kernel up to 5.10. The Git version up to 5.15 but this didn’t work for me. I couldn’t load ZFS anymore. I tried to repair everything one day but at the end i reinstalled my Pi.

Now running: 5.10.82

Config:config.txt KSMTuned Config:ksmtuned.txt

[Sysadminfromhell]

Little Update who compile the kernel with my config:

You have to recreate the ZFS and Ceph DKMS Modules!

dpkg-reconfigure zfs-dkms dpkg-reconfigure ceph-dkms

Dont`t panic if the ZFS Module doesnt load after you "installed" the Kernel, the Modules are missing for the 5.10.82 in the /var/lib/modules folders.

EDIT:

Prevent Kernel override from apt update/apt upgrade/apt dist-upgrade:

sudo apt-mark hold libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 sudo apt-mark hold raspberrypi-bootloader raspberrypi-kernel raspberrypi-kernel-headers

[aanon4]

Okay folks, I guess I'm going to finally put together a "official" pimox kernel. What requests do people have? Please assume I'm stupid and won't guess at what's wanted :-)

On my list currently:

KSM Included CEPH and RBD modules 9000 MTU

[Sysadminfromhell]

Maybe remove the unnessacery support for other devices like Gamepads. I dom not know if the Pi support it: SMT and Mutliparallel code execution for virtualisation, but this i do not know. I recommend you to stay on the 5.10. because of the ZFS Module which is included in the deb of debian, its only supporting up to Kernel 5.10.

[aanon4]

I'm starting from the basic raspberry pi kernel build (which is 5.10.83)

[Sysadminfromhell]

I'm starting from the basic raspberry pi kernel build (which is 5.10.83)

They updated the sources from .82 to .83. Good starting point

[Sysadminfromhell]

Okay folks, I guess I'm going to finally put together a "official" pimox kernel. What requests do people have? Please assume I'm stupid and won't guess at what's wanted :-)

On my list currently:

KSM Included CEPH and RBD modules 9000 MTU

I just recompiled my kernel, and have a few things which may could be useful:

General Setup -> Preemption Model (No Force Preemption (Server)) Kernel Features -> Enable paravirtualization Code CPU Power Management -> CPU Frequency scaling -> Default CPUFreq governor (onedmand) // instead of Powersave Firmware Drivers -> QEMU fw_cfg device support in sysfs (as Module) Memory Management options -> Free page reporting Memory Management options -> Enable bounce buffers Memory Management options -> Enable KSM for page merging Memory Management options -> compressed cache for swap pages Compressed cache for swap pages default compressor (LZO) Compressed cache for swap pages default allovator (zbud) Device Drivers -> Virtualization drivers Device Drivers -> virtio drivers Device Drivers -> Virtio drivers -> PCi driver for virtio devices (as Module) Device Drivers -> Virtio drivers -> Virtio balloon driver (as Module) Device Drivers -> Virtio drivers -> Virtio input driver (as Module) Device Drivers -> VHOST driver Device Drivers -> VHOST driver -> Host kernel accelerator for virtio net (as Module) Device Drivers -> VHOST driver -> vhost virtio-vsock driver (as Module) Device Drivers -> VHOST driver -> Cross-endian support for vhost

This may should someone Test before:

Kernel Features -> Multi-core scheduler support Kernel Features -> SMT scheduler support

Thats what I did in mine, just for a start maybe. I didnt remove anything so far, because I dont have a test-pi (out of stock in my country)

Kind regards,

[aanon4]

Well .. that took longer than I'd hoped, but here is a kernel to finally try. To install type (as root):

apt update
apt-mark hold libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0
apt-mark hold raspberrypi-bootloader raspberrypi-kernel raspberrypi-kernel-headers
apt -o Dpkg::Options::="--force-overwrite" install pve-kernel-5.10.83-1-pve

A few notes:

First, not sure what the appropriate way to deal with the apt-marks and the --force-overwrite options are. This package currently goes over the standard kernel package. It might be possible to just replace it at some point but for now it just overwrites.

Second, this is based of the Proxmox kernel package so it comes with zfs included (and ceph drivers) so the dkms packages can be removed. Again, not sure how to do this in Debian yet.

Third, I think it contains the wish list above but let me know what else might be useful.

Fourth, I have an 9000 MTU fix in (but not the one floating about on the Internet which messes with the defaults for all network devices). My testing is inconclusive though and I think I have an issue with my switch. I have a new one on the way (by coincidence) so will test again once that arrives.

[Sysadminfromhell]

Someone tried the kernel yet?

[mietzen]

I've tried, after this patch I can't start containers anymore:

run_apparmor_parser: 919 Failed to run apparmor_parser on "/var/lib/lxc/100/apparmor/lxc-100_<-var-lib-lxc>": apparmor_parser: Unable to replace "lxc-100_</var/lib/lxc>".  Profile does not conform to protocol
apparmor_prepare: 1089 Failed to load generated AppArmor profile
lxc_init: 850 Failed to initialize LSM
__lxc_start: 2007 Failed to initialize container "100"
TASK ERROR: startup for container '100' failed

@aanon4 this should revert all changes, or did I forget something?

apt-mark unhold $(apt-mark showhold)
apt update
apt remove pve-kernel-5.10.83-1-pve
apt install raspberrypi-kernel --reinstall
apt autoremove
dpkg-reconfigure zfs-dkms

Edit: forgot to rebuild dkms

[mietzen]

New Update: i had to rollback to the 5.10 because of the ZFS-DKMS Module. The ZFS-DKMS from the official DEB only support kernel up to 5.10. The Git version up to 5.15 but this didn’t work for me. I couldn’t load ZFS anymore. I tried to repair everything one day but at the end i reinstalled my Pi.

Now running: 5.10.82

Config:config.txt KSMTuned Config:ksmtuned.txt

I now compiled the kernel following your description and it works like a charm 👍

[Sysadminfromhell]

New Update: i had to rollback to the 5.10 because of the ZFS-DKMS Module. The ZFS-DKMS from the official DEB only support kernel up to 5.10. The Git version up to 5.15 but this didn’t work for me. I couldn’t load ZFS anymore. I tried to repair everything one day but at the end i reinstalled my Pi.

Now running: 5.10.82

Config:config.txt KSMTuned Config:ksmtuned.txt

I now compiled the kernel following your description and it works like a charm 👍

Do you have better performance with 5.10.90 ? You feel something ? I'm thinking off recompile my 5.10.82 config to make a new 5.10.90.

[mietzen]

Sorry I've no comparison to 5.10.82, but I can't image it will make a difference since it 'only' has a few more patches. But compared to a kernel without KSM, it's smoother (feeling). Btw. was my first time compiling / cross-compiling the kernel, I followed this guide: https://github.com/geerlingguy/raspberry-pi-pcie-devices/tree/master/extras/cross-compile Took me only 6 Minutes to compile and install 👍 I thought it would be much more of a hassle 😉

[aanon4]

I assume these kernels have apparmor disabled by default? if you disable it in /boot/cmdline.txt (apparmor=0) then containers are fine ... if a little more exposed than is perhaps ideal.

[aanon4]

It seems my version of the kernel doesn't support the network apparmor feature - not sure why. Here's a quick diff to remove the problematic feature in the configuration:

*** /usr/share/apparmor-features/features.orig  2022-01-13 18:48:15.308174008 -0800
--- /usr/share/apparmor-features/features   2022-01-13 19:20:07.296730382 -0800
***************
*** 31,41 ****
  mount {mask {mount umount pivot_root
  }
  }
- network {af_unix {yes
- }
- af_mask {unspec unix inet ax25 ipx appletalk netrom bridge atmpvc x25 inet6 rose netbeui security key netlink packet ash econet atmsvc rds sna irda pppox wanpipe llc ib mpls can tipc bluetooth iucv rxrpc isdn phonet ieee802154 caif alg nfc vsock kcm qipcrtr smc
- }
- }
  file {mask {create read write exec append mmap_exec link lock
  }
  }
--- 31,36 ----

[mietzen]

I assume these kernels have apparmor disabled by default? if you disable it in /boot/cmdline.txt (apparmor=0) then containers are fine ... if a little more exposed than is perhaps ideal.

Apparmor is loaded but not active:

# apparmor_status
apparmor module is loaded.
apparmor filesystem is not mounted.

[Sysadminfromhell]

You have to activate it via /boot/cmdline.txt by adding at the end: lsm="apparmor" i tested it, and it worked for me in my kernel.

https://forums.raspberrypi.com/viewtopic.php?t=66748 https://github.com/raspberrypi/linux/commit/ceea6dc03e90c230495227eb6f267ad1dd388730

[Sysadminfromhell]

BEWARE: LXC Contrainers doesnt work anymore because the profile parser fails on lxc-container-start cant be parsed. i deactivated apparmor for me and removed the apparmor package bc i do not need it. What is apparmor anyways?

[varoudis]

@Sysadminfromhell Any .config for pi4 available?

Reading the thread Im not sure if the working version is 5.10.90 (I just cross compiled it with the defconfig setting from pi foundation just as a compile test) or the later 5.15 etc.

[benep]

I had the apparmor problem on RockPi4 with armbian but I found a workaround:

nano /etc/pve/lxc/100.conf
lxc.apparmor.profile: lxc-default-with-nesting

[varoudis]

Hi @aanon4 Whats the best way to update/upgrade an existing pimox after you have pushed updates to github? Thanks

[aanon4]

I try to up date the repo monthly, so the standard Proxmox update mechanism should work for you.

[varoudis]

I was really afraid to do a GUI update/upgrade (as I do with the x86 proxmox I have). Thanks a lot!

[Sysadminfromhell]

Well .. that took longer than I'd hoped, but here is a kernel to finally try. To install type (as root):

apt update
apt-mark hold libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0
apt-mark hold raspberrypi-bootloader raspberrypi-kernel raspberrypi-kernel-headers
apt -o Dpkg::Options::="--force-overwrite" install pve-kernel-5.10.83-1-pve

A few notes:

First, not sure what the appropriate way to deal with the apt-marks and the --force-overwrite options are. This package currently goes over the standard kernel package. It might be possible to just replace it at some point but for now it just overwrites.

Second, this is based of the Proxmox kernel package so it comes with zfs included (and ceph drivers) so the dkms packages can be removed. Again, not sure how to do this in Debian yet.

Third, I think it contains the wish list above but let me know what else might be useful.

Fourth, I have an 9000 MTU fix in (but not the one floating about on the Internet which messes with the defaults for all network devices). My testing is inconclusive though and I think I have an issue with my switch. I have a new one on the way (by coincidence) so will test again once that arrives.

did you Update to the new 5.10.92 already? I had to reinstall the Pi after i tried to change the Network to a different subnet. So I´m willing to try this kernel instead of compiling my own.

[hastmu]

Hi, so i am not after the features you are looking for, but i came across the same issue with zfs-2.0.3 and 5.10+. i got hit by the last update from rasbian as i use zfs the dkms failed as you know, i just downloaded zfs-2.0.7 compiled it and installed the modules. root@rp4-pve-0001:~# uname -a Linux rp4-pve-0001 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux root@rp4-pve-0001:~# zfs -V zfs-2.0.3-9 zfs-kmod-2.0.7-1

the binaries are from the 2.0.3 installation, up to now it works quite fine. maybe it helps you.