harden modules load broken

adrelanos commented 1 year ago

Using "normal" (default settings) kernel. Not VM kernel.

sudo journalctl -u harden-module-loading.service

Nov 05 22:44:57 host systemd[1]: Starting harden-module-loading.service - Disable the loading of modules to the kernel after startup. 
Nov 05 22:44:57 host disable-kernel-module-loading[4406]: kernel.modules_disabled = 1
Nov 05 22:44:57 host disable-kernel-module-loading[4404]: The loading of new modules to the kernel has been disabled by security-misc
Nov 05 22:44:57 host systemd[1]: harden-module-loading.service: Deactivated successfully.
Nov 05 22:44:57 host systemd[1]: Finished harden-module-loading.service - Disable the loading of modules to the kernel after startup.

Ok, so it says it's done. But this contradicts it:

cat /proc/sys/kernel/modules_disabled

0

sysctl kernel.modules_disabled

kernel.modules_disabled = 0

But maybe sysctl only looks at configs, not at actual kernel values.

adrelanos commented 1 year ago

Also broken in non-Qubes. Broken in Kicksecure (for VirtualBox but I doubt VirtualBox is related).

sudo systemctl status harden-module-loading.service
○ harden-module-loading.service - Disable the loading of additional modules after systemd-modules-load.service
     Loaded: loaded (/lib/systemd/system/harden-module-loading.service; enabled; preset: enabled)
     Active: inactive (dead)

Nov 06 00:52:42 localhost systemd[1]: sysinit.target: Found ordering cycle on harden-module-loading.service/start
Nov 06 00:52:42 localhost systemd[1]: sysinit.target: Found dependency on sysinit.target/start
Nov 06 00:52:42 localhost systemd[1]: sysinit.target: Job harden-module-loading.service/start deleted to break ordering cycle starting with sysinit.target/start
Nov 06 00:52:42 localhost systemd[1]: sysinit.target: Found ordering cycle on harden-module-loading.service/start
Nov 06 00:52:42 localhost systemd[1]: sysinit.target: Found dependency on sysinit.target/start
Nov 06 00:52:42 localhost systemd[1]: sysinit.target: Job harden-module-loading.service/start deleted to break ordering cycle starting with sysinit.target/start

adrelanos commented 1 year ago

The systemd unit was fixed but this breaks the boot process of Kicksecure. This is because fewer modules are loaded.

This command is handy to get a sorted, deterministic list of modules:

lsmod | tail -n +2 | cut -d ' ' -f 1 | sort

I'll add this to helper-scripts as lsmod-deterministic. Will be useful during development.

adrelanos commented 1 year ago

https://github.com/Kicksecure/helper-scripts/blob/master/usr/bin/lsmod-deterministic

That script is of course optional.

adrelanos commented 1 year ago

For development, to check the influence of what kernel modules changed...

Disable the harden modules load systemd unit.

sudo systemctl disable harden-module-loading.service

Reboot.

Get a list of loaded kernel modules.

lsmod-deterministic > lsmod1

Enable this.

sudo systemctl disable harden-module-loading.service

Reboot.

Get another list of loaded kernel modules.

lsmod-deterministic > lsmod2

Compare.

meld lsmod1 lsmod2

adrelanos commented 1 year ago

It has some of the same issues listed here: https://www.kicksecure.com/wiki/Operating_System_Hardening#Module_Loading

Some of the same modules aren't load.

I think this won't be easy. Therefore meanwhile I'll disable this systemd unit with a preset. (It can still be easily manually enabled for development.)

adrelanos commented 1 year ago

It has some of the same issues listed here:

"Same issues" might sound not very severe. I should have said Xfce no longer starts. Once I was able to switch to a virtual terminal (VT) (to get the list of loaded modules to compare) but then after a reboot not again. So this is very severe and completely breaking the system.

monsieuremre commented 1 year ago

Oh I see. I did all the testing on debian. I also did test it on Kicksecure and had no problems, in a KVM VM. What might be the cause? What extra modules does kicksecure/whonix load to the kernel, and more precisely, when? If it loads its own modules with a service, they have to be added as a After= in this service's config.

On a native bare-metal debian and on kicksecure KVM, I can't reproduce any of the problems. Not tested on whonix. Not tested on Virtualbox. Not tested on Qubes or alike.

For virtual box, I already expected it to break, but not when we are the guest os. It would break when we do the virtualization of course, because they want to stick their own stuff to out pure protected kernel. That's why it also breaks when secure boot is on. But it doesn't make that sense if it breaks if the virtualized os has this hardening. Weird.

It is not surprising for Qubes to break either to be honest because it breaks on a daily basis if anything at all happens. But I think we might fix it if we find the root. Is it about Qubes or is it about Xen that causes the issue?

adrelanos commented 1 year ago

Oh I see. I did all the testing on debian. I also did test it on Kicksecure and had no problems, in a KVM VM. What might be the cause?

It breaks kernel on-demand module loading.

What extra modules does kicksecure/whonix load to the kernel, and more precisely, when?

Undefined. It's automatic, implicit. Based on-demand module loading. There are no manual modprobe invocations.

evdev, drm (Direct Rendering Manager) (not Digital Restrictions Management) kernel modules seem to be requires for Xfce (inside VirtualBox) loaded based on on-demand module loading. Also uinput (required by kloak - loaded through on-demand module loading). (https://github.com/vmonaco/kloak/issues/16)

For virtual box, I already expected it to break, but not when we are the guest os. It would break when we do the virtualization of course, because they want to stick their own stuff to out pure protected kernel.

For VirtualBox I could find a solution to hardcode the modules so this isn't the biggest of my worries kernel on-demand module loading broken is the core issue which is huge.

That's why it also breaks when secure boot is on.

Nowadays VirtualBox is compatible with Secure Boot enabled, I think, at least if enrolling the DKMS signing key into the MOK. This improved a lot in Debian since Debian bookworm also thanks to DKMS.

Is it about Qubes or is it about Xen that causes the issue?

No idea.

For this to be enabled by default, the contributor would need to make sure this is ready for most commonly used workflows (desktop, server, Whonix Xfce, Qubes) without breaking the boot process.

I could make a very specific list of kernel modules that aren't load but I don't see how that helps since the bulk of the implementation work needs to be contributed and I don't think that is possible without actual testing on these platforms anyhow. Only having that list wouldn't be very useful since it moves no step closer to fixing the issue of on-demand module loading.

https://github.com/systemd/systemd/issues/13540 doesn't sound like this ticket can be implements for a general purpose Linux distribution by default.

Following the instructions https://www.kicksecure.com/wiki/Operating_System_Hardening#Module_Loading will has a good chance fix the boot process but it's a very limited, very specific solution. Hardcoding all of these modules somewhere in default shipped config files (per package) seems wrong, high maintenance effort, difficult to debug.

monsieuremre commented 1 year ago

I mean, some of these were also given. We disable on-demand module loading. Loading is only allowed during boot. Normally on boot we pretty much load only what is needed. When you plug in your cool peripherals, they won't work as well, unless they were plugged in when starting up. So, what I think is, the absolute one and only solution here is just secure boot, I guess. Which should be compatible with anything that came out in the last 15 years. I'm not sure if it would be possible to get a similar functionality on legacy bios.

adrelanos commented 1 year ago

monsieuremre:

I mean, some of these were also given. We disable on-demand module loading. Loading is only allowed during boot. Normally on boot we pretty much load only what is needed. When you plug in your cool peripherals, they won't work as well, unless they were plugged in when starting up.

That won't work either. For example, kloak needs 1 kernel module. But it won't be load as this PR is currently implemented because kloak systemd unit starts later when module loading got already disabled.

Applications / systemd units that run later and use on-demand kernel module loading generally won't work.

So, what I think is, the absolute one and only solution here is just secure boot, I guess.

How does that help?

Btw EFI based Kicksecure / Whonix VM images might be coming soon thanks to https://github.com/grml/grml-debootstrap/issues/225 and then Secure Boot might not be far away either. ...if there is a solution for the out-of-tree tirdad kernel module.

Let's see. If you like to advocate for either legacy boot or EFI boot for VM images, please open a new issue.

Which should be compatible with anything that came out in the last 15 years. I'm not sure if it would be possible to get a similar functionality on legacy bios.

Sounds like https://github.com/Kicksecure/security-misc/issues/156

monsieuremre commented 1 year ago

because kloak systemd unit starts later

That looks a very easy to solve problem. Why not just add After=kloak.service?

adrelanos commented 1 year ago

That looks a very easy to solve problem. Why not just add After=kloak.service?

Because that does not seem to be a very clean solution.
Would be a very specific, not a generic solution and would only fix kloak, none of the other mentioned issues.
Does not seem to belong into security-misc.
It might cause a systemd dependency cycle (untested).
That might not scale. In the future, we'd add another After= entry to this systemd unit?
- I'd be more inclined to add file /usr/lib/modules-load.d/kloak.conf to the kloak package with content uinput.
Where does that lead... Then what's next... Package desktop-config-dist shipping file /usr/lib/modules-load.d/xfce.conf with content:

evdev
drm

So we'd do a completely hardcoded kernel module definition?

Then we wouldn't even need the new systemd unit because it fixes none of the previous issues. The list of files for /usr/lib/modules-load.d folder with what modules to add there was already researched and documented in the past: https://www.kicksecure.com/wiki/Operating_System_Hardening#Module_Loading

Might even still be functional.

I don't even see error messages in the systemd journal if modules fail to load. This is difficult to debug if other packages break.

Doesn't seem to be worth it. Not giving much security. Preventing new modules to be load is easily circumvented by malware writing to /usr/lib/modules-load.d or similar and just 1 reboot away.

Wouldn't the untrusted root concept if implemented cover this?

monsieuremre commented 1 year ago

Of course. Let alone untruted root, my policy did cover this. The one you deleted. More on that in a bit.

faelmori commented 2 months ago

Have you tried something like this:

first upgrade the stuff: upgrade-nonroot

enable the service: sudo systemctl enable --force --now harden-module-loading.service

validate the mokutil installation with: sudo apt install --no-install -recommends -yyq mokutil

import the dkms key with: sudo mokutil --import /var/lib/dkms/mok.pub (keep the password)

reboot to finalize/register your BIOS: sudo reboot (put same password as in the last step when the BIOS starts the key registration process)

then, if you still see errors loading kernel modules due to signature approval errors, reinstall VirtualBox using the whonix script (already available in kicksecure): for xfce: whonix-xfce-installer-cli --log-level=info

for others : whonix-cli-installer-cli --log-level=info

adrelanos commented 2 months ago

This is not a secure boot / kernel module verification issue.

Kicksecure / security-misc

harden modules load broken #159