LKRG performs runtime integrity checking of the Linux kernel and detection of security vulnerability exploits against the kernel.
LKRG is a kernel module (not a kernel patch), so it can be built for and loaded on top of a wide range of mainline and distros' kernels, without needing to patch those. We currently support kernel versions ranging from as far back as RHEL7's (and its many clones/revisions) and Ubuntu 16.04's to latest mainline and distros' kernels. Our Continuous Integration setup has tested this version of LKRG with up to latest mainline kernel 6.8.0-060800rc6daily20240227-generic as available for Ubuntu on the release date.
LKRG currently supports the x86-64, 32-bit x86, AArch64 (ARM64), and 32-bit ARM CPU architectures.
Please refer to CONCEPTS for concepts behind LKRG and for information on its efficacy, and to PERFORMANCE for information on its performance impact.
The following sections describe how to obtain LKRG sources, build LKRG, test it, install it on the system, and customize its configuration.
For LKRG releases and latest source code, please refer to its homepage:
https://lkrg.orgTo download this release from there and verify it, you would have used commands like the below:
wget https://www.openwall.com/signatures/openwall-offline-signatures.asc
gpg --import openwall-offline-signatures.asc
wget https://lkrg.org/download/lkrg-0.9.8.tar.gz.sign
wget https://lkrg.org/download/lkrg-0.9.8.tar.gz
gpg --verify lkrg-0.9.8.tar.gz.sign lkrg-0.9.8.tar.gzPlease preserve the GnuPG key above and also use it to verify future releases, which will most likely work in a similar manner.
Latest LKRG development source code is hosted on GitHub, from where you can clone the git repository to a local directory using the following command:
git clone https://github.com/lkrg-org/lkrgTo build LKRG, you will need the following software:
GNU make
GCC, ideally the same version of it that was used to build the kernel itself (some people manage with clang, but this is unsupported, so expect issues)
libelf, including its "development" sub-package, in case your target kernel was built with CONFIG_UNWINDER_ORC=y
A kernel build directory corresponding to the Linux kernel image the module is to run on.
For example, under Debian and Ubuntu you can install all of these with:
sudo apt-get install make gcc libelf-dev linux-headers-$(uname -r)and under Red Hat'ish distributions (e.g. RHEL, CentOS, Fedora) with:
sudo yum install make gcc elfutils-libelf-devel kernel-devel(For documentation purposes, we prefix commands requiring root access with "sudo", but you may of course run them as root by different means.)
With the above requirements satisfied, you should be able to easily build LKRG by running "make" when you're in LKRG's top level source code directory. Building LKRG does not require root, and thus shouldn't be done as root.
To speed up the building, we recommend specifying a parallel job count matching your machine's logical CPU count, e.g. like this:
make -j8We recommend that before you install LKRG on the system such that it would be started on bootup, you manually test loading the LKRG module into the kernel without making the setup permanent. We also recommend that you keep LKRG's detection of kernel integrity violations enabled for this test, yet change its enforcement action from kernel panic (the default) to mere logging. This way, you can safely detect potential system-specific false positives and only proceed with installation if there are none.
You can do this for a freshly built LKRG (and while you're still in its top level source code directory) with the following command:
sudo insmod output/lkrg.ko kint_enforce=1Then check kernel messages for any potential errors, use the system for a long while, and check again:
sudo dmesg(Depending on kernel version and system configuration, the "dmesg" command might not require root.)
Unload LKRG from the kernel with:
sudo rmmod lkrgso that it can then be loaded using the same procedure that's used on system bootup and without the parameter override.
If your Linux distribution uses a supported init system (systemd or OpenRC), you can install LKRG with:
sudo make installwhile you're still in its top level source code directory.
We don't in any way favor one init system over another, and would gladly add support for more of them if there's demand, or especially if we receive such contributions. Meanwhile, on a distribution without a supported init system you can let "sudo make install" partially complete (up to the point where it finds you're not using a supported init system).
Run the following command to start the LKRG service, for systemd:
sudo systemctl start lkrgfor OpenRC:
sudo /etc/init.d/lkrg startfor other:
sudo modprobe -v lkrgIn order to automatically load LKRG into the Linux kernel on each bootup run the following command, for systemd:
sudo systemctl enable lkrgfor OpenRC:
sudo rc-update add lkrg bootfor other:
sudo mkdir -p /etc/modules-load.d/ &&
echo lkrg | sudo tee /etc/modules-load.d/lkrg.confAlternatively, you can put the "modprobe lkrg" command into a system startup script. Please note that ideally this command would run before sysctl files (especially /etc/sysctl.d/01-lkrg.conf) are processed, or otherwise the LKRG settings specified in those would not take effect.
DKMS enables kernel modules to be dynamically built for each kernel version. What this means in effect is that on kernel upgrades the module is rebuilt. You can install LKRG using DKMS as well. For instance, on Red Hat'ish distributions after following the shared download instructions above:
sudo tar -xzf lkrg-0.9.8.tar.gz -C /usr/src/
sudo dnf update -y
sudo dnf install kernel-devel dkms openssl
sudo dkms add -m lkrg -v 0.9.8
sudo dkms build -m lkrg -v 0.9.8
sudo dkms install -m lkrg -v 0.9.8The only difference on other distributions should be the installation of the kernel headers, the DKMS utility, and OpenSSL. Install the headers for the target kernels.
You can then query the status with:
dkms statusIf everything is right, you should get similar output to the following:
lkrg/0.9.8, 5.18.9-200.fc36.x86_64, x86_64: installedPlease refer to the previous two sections for how to start the LKRG service or
have it started on system bootup. If you wish to use the unit/init file, you
must install it manually, e.g., by running the lkrg-bootup.sh script
located under scripts/bootup/ with the install subcommand (as root).
Similarly to installation, you can uninstall LKRG using "make" as well:
sudo make uninstallwhile you're in the top level source code directory of the installed version.
If you installed using DKMS, you'd uninstall with:
sudo dkms remove -m lkrg/0.9.8 --allYou can also use the following command to temporarily stop the LKRG service without uninstalling it, for systemd:
sudo systemctl stop lkrgfor OpenRC:
sudo /etc/init.d/lkrg stopfor other:
sudo modprobe -v -r lkrgOur suggested way to upgrade LKRG is to start by uninstalling the old version.
You can then follow the Testing and Installation steps for the new version.
To account for the hopefully unlikely, but really unfortunate event that some incompatibility between the Linux kernel or other components of the system and LKRG isn't detected prior to LKRG installation, yet leads to system crash on bootup, we've included support for the "nolkrg" kernel parameter. Thus, you may disable LKRG by specifying "nolkrg" on the kernel command-line via your bootloader. The system should then boot up without LKRG, and thus without triggering the problem, letting you fix it. You must be aware though, that you will not be able to manually load the LKRG module if the kernel was booted with this parameter.
The LKRG kernel module supports a number of parameters, including kint_enforce already mentioned above and many more.
For freshly built LKRG, you can list the parameters with:
modinfo output/lkrg.kowhile you're still in LKRG's top level source code directory.
With LKRG installed on the system, you can list them with:
sudo modinfo lkrg(Depending on system configuration, "modinfo" might not require root.)
Parameters can be specified on command-lines of "insmod", "modprobe", or after "options lkrg " in a file in the /etc/modprobe.d directory.
For descriptions of the parameters and their default and possible values, please refer to the following two sections.
LKRG supports the following module parameters (with default values or lack thereof specified in braces) to enable its optional remote logging.
net_server_addr (no default) Log server IPv4 address (e.g., 127.0.0.1)
net_server_port (514) Log server TCP port number
net_server_pk (no default) Log server public key (64 hexadecimal digits)
If you're starting LKRG via a systemd unit or startup script (such as those provided in here), our recommended way to specify the above parameters is by creating the file /etc/modprobe.d/lkrg.conf with something like this in it:
options lkrg net_server_addr=127.0.0.1 net_server_pk=64hexdigitshere
Please refer to LOGGING on how to use the corresponding userspace components.
Besides the parameters optionally specified when loading the module into the kernel, LKRG also supports a number of sysctl's, which can be used to adjust its behavior when it is already loaded into the kernel. For each feature that is configurable at both load time and run time, we have a module parameter and a sysctl of similar name (the module parameters lack the "lkrg." prefix, but are otherwise the same), so the below documentation is mostly usable for both.
To list all LKRG sysctl's and their current values, use:
sudo sysctl -a | grep lkrgThe sysctl's are (with default values specified in braces):
lkrg.profile_validate (3) Quick choice of a pre-defined profile controlling whether, when, and to what extent LKRG validates system integrity and detects attacks. Allowed values are 0 (disabled), 1 (light), 2 (balanced), 3 (heavy), and 4 (paranoid). Additionally, this setting will read as 9 (custom) if an underlying setting is changed directly (potentially deviating from any of the profiles).
Higher-numbered validation profiles provide higher likelihood of timely detection of an attack, but involve higher performance overhead and higher risk of incompatibility with other system software. Profiles 1 to 3 provide reasonable tradeoffs.
lkrg.profile_validate=3 or higher is incompatible with VirtualBox hosts, where you need to use at most lkrg.profile_validate=2. However, there's no problem with setting lkrg.profile_validate=3 on Linux+LKRG guest systems in VirtualBox VMs.
lkrg.profile_validate=4 (paranoid) is incompatible with many distributions and has unreasonably high performance overhead and poor scalability while not necessarily providing a practically relevant improvement in attack detection.
Choosing a validation profile sets the following underlying settings, which are described further below: kint_validate, pint_validate, pcfi_validate, umh_validate, smep_validate, smap_validate, and msr_validate.
lkrg.profile_enforce (2) Quick choice of a pre-defined profile controlling whether and how LKRG acts on detected integrity violations and attacks. Allowed values are 0 (log and accept), 1 (selective), 2 (strict), and 3 (paranoid). Additionally, this setting will read as 9 (custom) if an underlying setting is changed directly (potentially deviating from any of the profiles).
Higher-numbered enforcement profiles provide higher likelihood of mitigating a compromise or stopping an attack, but also a higher risk of interfering with normal system behavior and to a worse extent in case of false positives.
lkrg.profile_enforce=0 can be used for safe testing of LKRG, where any detected violations and attacks are logged but no enforcement is performed. It can also be useful where LKRG is meant to act as a sensor within a larger security monitoring and response setup (e.g., network-wide).
lkrg.profile_enforce=1 performs selective enforcement - log only for kernel integrity violations, varying effective actions ranging from killing a task to triggering a kernel panic for other types of violations and attacks. This mode is extremely unlikely to panic the kernel on a false positive.
lkrg.profile_enforce=2 performs strict enforcement - varying effective actions for all types of violations and attacks, including triggering a kernel panic for kernel integrity violations.
lkrg.profile_enforce=3 performs the most paranoid enforcement - kernel panic for all types of violations and attacks.
Choosing an enforcement profile sets the following underlying settings, which are described further below: kint_enforce, pint_enforce, pcfi_enforce, umh_enforce, smep_enforce, and smap_enforce.
Also relevant is the kernel's kernel.panic sysctl and panic parameter, which makes the system reboot on kernel panic. For example, kernel.panic=60 in /etc/sysctl.conf or in a file under the /etc/sysctl.d directory, or panic=60 on the kernel's command-line, will make the system reboot in 60 seconds after a panic. This provides a brief opportunity to read the panic message on the console yet makes an unattended server try to come back up on its own.
Profiles are currently available via sysctl only - there are no corresponding module parameters. However, the individual underlying settings, which are described further below, do have their corresponding module parameters.
lkrg.heartbeat (0) Whether or not to print a heartbeat message ("System is clean!" or "Tasks are clean!" depending on other configuration) whenever the global integrity checking routine completes with no violations detected. Allowed values are 0 (don't print the message) and 1 (print the message if allowed by log_level).
lkrg.interval (15) LKRG's timer interval for periodic invocation of the global integrity checking routine, in seconds. Allowed values are 5 to 1800.
lkrg.trigger (N/A) Force LKRG to invoke the global integrity checking routine. If you set this to 1, the routine is immediately invoked and this sysctl is reset back to 0.
lkrg.log_level (3) LKRG's logging verbosity level. Allowed values are from 0 to 4 for normal builds or from 0 to 6 for debugging builds.
Values of 4 and higher are meant for debugging only and produce too verbose logging for production use. Moreover, some messages logged at those high levels contain information useful for kernel vulnerability exploitation, making those log levels potentially mildly insecure (depending on other system configuration).
lkrg.block_modules (0) Whether or not to block further loading of kernel modules. Allowed values are 0 (no) and 1 (yes).
This feature is meant primarily to prevent unintended user-triggered (or attacker-triggered) auto-loading of maybe-vulnerable modules provided in a distribution after all intended modules have already been loaded. This feature is not effective (nor is meant to be) against attackers who already have root privileges and try to load a module explicitly (they could simply flip this setting or even unload LKRG first).
Please note that enabling this setting (too) early (e.g., using the module parameter or /etc/sysctl.*) may cause the system to fail to complete bootup (if required modules are still being loaded in later stages of bootup, which varies between distributions and system configurations).
Also relevant is the kernel's kernel.modules_disabled sysctl, which fully disables module loading until the system is rebooted.
lkrg.hide (0) Whether or not LKRG should hide itself from the lists of loaded modules and KOBJs. Allowed values are 0 (do not hide LKRG, or unhide it if previously hidden) and 1 (hide LKRG).
Please note that LKRG can be easily detected by other means anyway, such as through the presence of its sysctl's.
lkrg.kint_validate (3) Whether and when to validate global kernel integrity. Allowed values are 0 (disabled), 1 (only when manually triggered by lkrg.trigger), 2 (also periodically every lkrg.interval seconds), and 3 (also periodically every lkrg.interval seconds and probabilistically on certain other events).
This currently applies to kernel and modules code and read-only data, global SELinux settings, and some CPU status registers/bits (WP, SMEP, SMAP, MSRs). (The validation and enforcement of SMEP, SMAP, and MSRs are separately controlled by their respective knobs described below, and SMEP and SMAP are validated much more frequently, not only as part of global kernel integrity.)
lkrg.kint_enforce (2) How to act on global kernel integrity violations. Allowed values are 0 (log once and accept new likely-compromised state as valid), 1 (log only for most violations, log the violation and restore previous state for SELinux and CPU WP bit), and 2 (panic the kernel).
Note that lkrg.kint_enforce=1 is expected to produce repeated log messages on most kernel integrity violations, which can be noisy. Also note that lkrg.kint_enforce=2 is unfortunately the only way to make full use of LKRG's global kernel integrity validation. Running with lkrg.kint_validate=2 or higher but lkrg.kint_enforce set to 0 or 1 wastes CPU time on costly checks without achieving a corresponding security improvement, except that it might provide logs for post-mortem detection and analysis of a security compromise.
lkrg.pint_validate (2) Whether and when to validate process credentials integrity. Allowed values are 0 (disabled), 1 (validate a task's credentials just before it'd make use of the credentials), 2 (currently, it has the same meaning as 1), and 3 (validate credentials of all tasks in the system whenever any task is about to make use of its credentials).
Except with lkrg.pint_validate=0, we also validate the credentials of all tasks as part of LKRG's global integrity checking routine.
lkrg.pint_validate=1 is sufficient to provide most of LKRG's potential at timely detection of exploits. lkrg.pint_validate=3 is a paranoid mode with high performance overhead yet likely a minuscule gain in security.
lkrg.pint_enforce (1) How to act on process credentials integrity violations. Allowed values are 0 (log once and accept new likely-compromised state as valid), 1 (kill the task), and 2 (panic the kernel).
In Linux kernel's terminology, which we also use here, a "task" refers to a thread, and threads of a program may technically have different credentials. Our enforcement of process credentials integrity is thus per-thread, and e.g. it might happen that we kill an individual compromised thread of a program.
lkrg.pcfi_validate (2) Whether and to what extent to validate Control Flow Integrity (CFI) on kernel functions that we monitor because of their usefulness for exploits' Return Oriented Programming (ROP) chains. Allowed values are 0 (disabled), 1 (only validate the stack pointer), and 2 (also validate all stack frames).
Because of the very limited extent of validation performed, we call our CFI mechanism pCFI, for poor man's CFI.
lkrg.pcfi_validate=2 is incompatible with VirtualBox hosts, where you need to use at most lkrg.pcfi_validate=1. However, there's no problem with setting lkrg.pcfi_validate=2 on Linux+LKRG guest systems in VirtualBox VMs.
lkrg.pcfi_enforce (1) How to act on pCFI violations. Allowed values are 0 (log only), 1 (kill the task), and 2 (panic the kernel).
Note that lkrg.pcfi_enforce=0 may produce repeated log messages for the same violation, which might occasionally be noisy.
lkrg.umh_validate (1) Whether and to what extent to validate uses of usermodehelper (UMH). Allowed values are 0 (validation disabled), 1 (allow only previously known programs), and 2 (completely block UMH).
UMH can also be protected with pCFI regardless of this setting.
UMH is a kernel-internal interface, which the kernel uses to invoke programs such as /sbin/modprobe (to auto-load a module on demand) and many others. When left unrestricted, UMH is convenient for kernel vulnerability exploits.
lkrg.umh_enforce (1) How to act on UMH usage violations. Allowed values are 0 (log only), 1 (prevent execution), and 2 (panic the kernel).
lkrg.smep_validate (1) Whether or not to validate the Supervisor Mode Execution Protection (SMEP) bit on supporting x86-64 CPUs. Allowed values are 0 (no) and 1 (yes).
lkrg.smep_enforce (2) How to act on unexpected changes of the SMEP bit. Allowed values are 0 (log once and accept new likely-compromised state as valid), 1 (log the violation and restore original value), and 2 (panic the kernel).
lkrg.smap_validate (1) Whether or not to validate the Supervisor Mode Access Prevention (SMAP) bit on supporting x86-64 CPUs. Allowed values are 0 (no) and 1 (yes).
lkrg.smap_enforce (2) How to act on unexpected changes of the SMAP bit. Allowed values are 0 (log once and accept new likely-compromised state as valid), 1 (log the violation and restore original value), and 2 (panic the kernel).
lkrg.msr_validate (0) Whether or not to validate CPU Model Specific Registers (MSRs) as part of the global integrity checking routine. Allowed values are 0 (no) and 1 (yes).
This is currently specific to x86(-64) CPUs.
There are situations where such validation is undesirable, such as if you run LKRG on a host machine that manages VMs and dynamically reconfigures MSRs. This is known to be the case for KVM and VirtualBox hosts, where this setting needs to be disabled. However, there's no problem with enabling this setting on Linux+LKRG guest systems in VMs on those hosts, and indeed on systems that don't run KVM and VirtualBox.
That's all for now. Greetings from the LKRG team!