"_stext hash changed unexpectedly" when loading at boot on RHEL8

[solardiz]

For a package of LKRG 0.9.7 built with our RPM spec file here on a RHEL 8 rebuild (package built for Rocky 8, loaded on Alma 8), after systemctl enable lkrg and reboot, with an edit to /etc/sysctl.d/01-lkrg.conf to prevent the panic:

[    4.076077] LKRG: ALIVE: Loading LKRG
[    4.588367] LKRG: ISSUE: [kretprobe] register_kretprobe() for <ovl_dentry_is_whiteout> failed! [err=-22]
[    4.623828] LKRG: ISSUE: [kretprobe] register_kretprobe() for ovl_dentry_is_whiteout failed and ISRA / CONSTPROP version not found!
[    4.623834] LKRG: ISSUE: Can't hook 'ovl_dentry_is_whiteout'. This is expected when OverlayFS is not used.
[    4.832509] LKRG: ISSUE: IOMMU table can't be found (skipping)
[    4.849556] LKRG: ALIVE: LKRG initialized successfully
[    5.000069] LKRG: STATE: Changing 'kint_enforce' from 2 (PANIC) to 1 (LOG ONLY (For SELinux and CR0.WP LOG & RESTORE))
[    7.571959] LKRG: ALERT: DETECT: Kernel: _stext hash changed unexpectedly
[    7.577733] LKRG: ALERT: DETECT: Kernel: 1 checksums changed unexpectedly
[    9.027341] LKRG: ALERT: DETECT: Kernel: _stext hash changed unexpectedly
[    9.032777] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_conntrack
[    9.032814] LKRG: ALERT: DETECT: Kernel: Module list hash changed unexpectedly
[    9.032844] LKRG: ALERT: DETECT: Kernel: Module KOBJ list hash changed unexpectedly
[    9.032874] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_conntrack
[    9.032901] LKRG: ALERT: DETECT: Kernel: 5 checksums changed unexpectedly
[   19.947718] LKRG: ALERT: DETECT: Kernel: _stext hash changed unexpectedly
[   19.952714] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_nat
[   19.952743] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_conntrack
[   19.952774] LKRG: ALERT: DETECT: Kernel: Module list hash changed unexpectedly
[   19.952805] LKRG: ALERT: DETECT: Kernel: Module KOBJ list hash changed unexpectedly
[   19.952835] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_conntrack
[   19.952861] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_nat
[   19.952886] LKRG: ALERT: DETECT: Kernel: 7 checksums changed unexpectedly
[   35.311938] LKRG: ALERT: DETECT: Kernel: _stext hash changed unexpectedly
[   35.317569] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_nat
[   35.317614] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_conntrack
[   35.317664] LKRG: ALERT: DETECT: Kernel: Module list hash changed unexpectedly
[   35.317738] LKRG: ALERT: DETECT: Kernel: Module KOBJ list hash changed unexpectedly
[   35.317796] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_conntrack
[   35.317831] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_nat
[   35.317867] LKRG: ALERT: DETECT: Kernel: 7 checksums changed unexpectedly
[   37.897693] LKRG: ALERT: DETECT: Kernel: _stext hash changed unexpectedly
[   37.903057] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_nat
[   37.903092] LKRG: ALERT: DETECT: Kernel: Module hash changed unexpectedly, name nf_conntrack
[   37.903123] LKRG: ALERT: DETECT: Kernel: Module list hash changed unexpectedly
[   37.903154] LKRG: ALERT: DETECT: Kernel: Module KOBJ list hash changed unexpectedly
[   37.903185] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_conntrack
[   37.903209] LKRG: ALERT: DETECT: Kernel: Module KOBJ hash changed unexpectedly, name nf_nat
[   37.903234] LKRG: ALERT: DETECT: Kernel: 7 checksums changed unexpectedly

and then these keep repeating. Unloading and reloading LKRG makes them go away. In my testing so far, this occurs only when LKRG is loaded at boot.

I think this is a different problem than #269 because this VM has only one vCPU (and is x86_64, but that might not be relevant).

[solardiz]

The above is with Linux 4.18.0-477.27.1.el8_8.x86_64 x86_64. Judging by the panics I got before I changed config, this is reproducible all the time. I haven't tried debugging it further yet.

[solardiz]

This issue does not occur so far on Rocky Linux 9 with Linux 5.14.0-284.30.1.el9_2.x86_64 x86_64 in a 4 vCPU VM. Same source package of LKRG, but obviously a different build of it (for 9).

[solardiz]

IIRC, an older version of LKRG worked correctly on 4.18.0-348.23.1.el8_5.x86_64, so we need to look for what changed between -348 and -477. Perhaps it's a back-port that we need to adapt our logic to (enable something that we already have for newer kernels, but did not yet enable for RHEL8 kernels).

[Adam-pi3]

Can you get the list of commit changes from Rocky Linux?

[solardiz]

The individual kernel commits are not exposed by RHEL anymore, but there's this in changelog:

* Mon Oct 03 2022 Lucas Zampieri <lzampier@redhat.com> [4.18.0-429.el8]
- x86/alternative: Relax text_poke_bp() constraint (Wander Lairson Costa) [2062175] {CVE-2021-26341}
- perf/x86: Add support for perf text poke event for text_poke_bp_batch() callers (Wander Lairson Costa) [2062175] {CVE-2021-26341}

* Thu Aug 04 2022 Jarod Wilson <jarod@redhat.com> [4.18.0-415.el8]
- x86/alternatives: Teach text_poke_bp() to emulate RET (Waiman Long) [2090229]
- x86/alternatives: Implement a better poke_int3_handler() completion scheme (Waiman Long) [2090229]
- x86/alternatives: Use INT3_INSN_SIZE (Waiman Long) [2090229]
- x86/kprobes: Fix ordering while text-patching (Waiman Long) [2090229]
- x86/kprobes: Convert to text-patching.h (Waiman Long) [2090229]
- x86/alternative: Shrink text_poke_loc (Waiman Long) [2090229]
- x86/alternative: Remove text_poke_loc::len (Waiman Long) [2090229]
- x86/ftrace: Use text_gen_insn() (Waiman Long) [2090229]
- x86/alternative: Add text_opcode_size() (Waiman Long) [2090229]
- x86/ftrace: Use text_poke() (Waiman Long) [2090229]
- x86/alternatives: Add and use text_gen_insn() helper (Waiman Long) [2090229]
- x86/alternatives, jump_label: Provide better text_poke() batching interface (Waiman Long) [2090229]

There were many more lines in these changelog entries - I kept here only those I thought were potentially relevant.

The issue looks similar to #256, and it appears that our fix for that one is wrong for recent RHEL 8. Indeed, it'd treat any 8.x the same as anything below 9.1, so we wouldn't adjust to a back-port only made e.g in 8.8.

Diff'ing -348 vs. -477, so 8.5 to 8.8, I see that ASM_RET and DISP32_SIZE have both appeared between these versions, and indeed struct text_poke_loc is of the kind we'd use on 9.1+, but the check for below 9.1 that we introduced when fixing #256 prevents us from using the right struct.

It is possible that the fix for #256 was buggy right away. Adam, you wrote that you "verified RHEL 8.6, 8.7, 9.0 and 9.1 and the current logic seems to work fine", but since we're dealing with races I'm not convinced nothing actually broke on 8.6 and 8.7. I don't know whether your testing included installation of LKRG to be loaded at system bootup?

[solardiz]

I generated these diffs between RHEL8 kernel revisions:

-rw-r--r--. 1 build build  86138405 Nov  8 18:45 348.23.1-to-372.32.1.diff
-rw-r--r--. 1 build build 284175212 Oct 26 19:19 348.23.1-to-477.27.1.diff
-rw-r--r--. 1 build build  89619660 Nov  8 19:06 372.32.1-to-425.19.2.diff
-rw-r--r--. 1 build build 123718534 Nov  8 19:06 425.19.2-to-477.27.1.diff

These correspond to the range from 8.5 to 8.8.

In there, text_poke_loc is mentioned in:

$ fgrep -l text_poke_loc *.diff
348.23.1-to-372.32.1.diff
348.23.1-to-477.27.1.diff
425.19.2-to-477.27.1.diff

In other words, it's possibly changed between 8.5 and 8.6, and then again between 8.7 and 8.8.

The 8.5 to 8.6 change removes struct text_poke_loc from arch/x86/include/asm/text-patching.h and adds a changed version of it to arch/x86/kernel/alternative.c:

--- x348.23.1/linux-4.18.0-348.23.1.el8_5/arch/x86/include/asm/text-patching.h  2022-04-12 15:30:37.000000000 +0200
+++ x372.32.1/linux-4.18.0-372.32.1.el8_6/arch/x86/include/asm/text-patching.h  2022-10-07 16:45:37.000000000 +0200
@@ -25,14 +25,6 @@ static inline void apply_paravirt(struct
  */
 #define POKE_MAX_OPCODE_SIZE   5

-struct text_poke_loc {
-       void *addr;
-       int len;
-       s32 rel32;
-       u8 opcode;
-       const u8 text[POKE_MAX_OPCODE_SIZE];
-};

--- x348.23.1/linux-4.18.0-348.23.1.el8_5/arch/x86/kernel/alternative.c 2022-04-12 15:30:37.000000000 +0200
+++ x372.32.1/linux-4.18.0-372.32.1.el8_6/arch/x86/kernel/alternative.c 2022-10-07 16:45:37.000000000 +0200
[...]
+struct text_poke_loc {
+       s32 rel_addr; /* addr := _stext + rel_addr */
+       s32 rel32;
+       u8 opcode;
+       const u8 text[POKE_MAX_OPCODE_SIZE];
+};

The 8.7 to 8.8 change is:

 struct text_poke_loc {
-       s32 rel_addr; /* addr := _stext + rel_addr */
-       s32 rel32;
+       /* addr := _stext + rel_addr */
+       s32 rel_addr;
+       s32 disp;
+       u8 len;
        u8 opcode;
        const u8 text[POKE_MAX_OPCODE_SIZE];
+       /* see text_poke_bp_batch() */
+       u8 old;
 };

[solardiz]

This appears fixed by #296, so I'll close. We can reopen if needed.