LKRG Trips kCFI During get_kallsyms_address (on-load)

[sempervictus]

Finally managed to get a 6.1.8 built with kCFI and LTO on LLVM16, LKRG still as a module. Unfortunately, the modules attempt to look-up addresses (it was built in-tree - in the hopes of avoiding this exact type of nonsense) at load-time produces this effect (hey, CFI works):

# modprobe lkrg 
[ 1285.469868][  T440] CFI failure at get_kallsyms_address+0xe6/0x150 [lkrg] (target: kallsyms_lookup_name+0x4/0x210; expected type: 0xaa7e236f)
[ 1285.471017][  T440] invalid opcode: 0000 [#1] PREEMPT SMP
[ 1285.471475][  T440] CPU: 3 PID: 440 Comm: modprobe Tainted: G                TN 6.1.8 #1
[ 1285.472168][  T440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.16.1-1-1 04/01/2014
[ 1285.472984][  T440] RIP: 0010:get_kallsyms_address+0xe6/0x150 [lkrg]
[ 1285.473570][  T440] Code: 28 48 89 05 f4 7c 02 00 48 89 e7 e8 44 e4 81 ec 4c 8b 1d e5 7c 02 00 48 c7 c7 41 06 9a c0 41 ba 91 dc 81 55 45 03 53 fc 74 02 <0f> 0b 41 ff d3 66 90 48 89 05 74 7d 02 00 31 db eb 10 48 c7 c3 ff
[ 1285.475287][  T440] RSP: 0018:ffffb18881bbf998 EFLAGS: 00010202
[ 1285.475863][  T440] RAX: 0000000000000000 RBX: ffffffffc0911005 RCX: 0000000000000000
[ 1285.476512][  T440] RDX: 0000000000384a83 RSI: 0000000000000000 RDI: ffffffffc09a0641
[ 1285.477155][  T440] RBP: ffffb18881bbfd70 R08: ffff97d9847ee0c0 R09: ffff97d980042800
[ 1285.477834][  T440] R10: 0000000055a0ebf7 R11: ffffffffad176a94 R12: ffffffffaea19728
[ 1285.478500][  T440] R13: 0000000000000000 R14: ffff97d9848e02a0 R15: 0000000000000000
[ 1285.479161][  T440] FS:  000066380b197740(0000) GS:ffff97dab7cc0000(0000) knlGS:0000000000000000
[ 1285.479924][  T440] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1285.480463][  T440] CR2: 000066380aaf4400 CR3: 000000010a1da001 CR4: 0000000000760ee0
[ 1285.481115][  T440] PKRU: 55555554
[ 1285.481396][  T440] Call Trace:
[ 1285.481654][  T440]  <TASK>
[ 1285.481894][  T440]  ? kallsyms_lookup_name+0x4/0x210
[ 1285.482309][  T440]  ? __cfi_p_tmp_kprobe_handler+0x10/0x10 [lkrg]
[ 1285.482873][  T440]  ? 0xffffffffc04c7000
[ 1285.483227][  T440]  ? __cfi_init_module+0x5/0x5 [lkrg]
[ 1285.483674][  T440]  init_module+0x17/0xffb [lkrg]
[ 1285.484105][  T440]  ? __cfi_init_module+0x5/0x5 [lkrg]
[ 1285.484554][  T440]  do_one_initcall+0x12d/0x2b0
[ 1285.484959][  T440]  ? rcu_nocb_gp_kthread+0x470/0x9f0
[ 1285.485384][  T440]  do_init_module+0x51/0x280
[ 1285.485776][  T440]  __se_sys_finit_module+0xae/0x100
[ 1285.486204][  T440]  do_syscall_64+0x76/0xb0
[ 1285.486575][  T440]  ? do_syscall_64+0x87/0xb0
[ 1285.486998][  T440]  ? syscall_exit_to_user_mode+0x32/0x160
[ 1285.487494][  T440]  ? do_syscall_64+0x87/0xb0
[ 1285.487914][  T440]  ? do_syscall_64+0x87/0xb0
[ 1285.488325][  T440]  ? exc_page_fault+0x66/0xc0
[ 1285.488750][  T440]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 1285.489276][  T440] RIP: 0033:0x66380ab19abd
[ 1285.489673][  T440] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9b 72 0d 00 f7 d8 64 89 01 48
[ 1285.491451][  T440] RSP: 002b:000077aba59dc228 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 1285.492218][  T440] RAX: ffffffffffffffda RBX: 00000305cddba1d0 RCX: 000066380ab19abd
[ 1285.492904][  T440] RDX: 0000000000000000 RSI: 00000305b664acb2 RDI: 0000000000000003
[ 1285.493591][  T440] RBP: 00000305b664acb2 R08: 0000000000000000 R09: 0000000000000000
[ 1285.494315][  T440] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000040000
[ 1285.495336][  T440] R13: 00000305cddba160 R14: 0000000000000000 R15: 00000305cddba4c0
[ 1285.496361][  T440]  </TASK>
[ 1285.496878][  T440] Modules linked in: lkrg(+) intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common nfit kvm_intel kvm irqbypass crct10dif_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel qxl rapl drm_ttm_helper ttm joydev cfg80211 mousedev drm_kms_helper tiny_power_button sysimgblt evdev syscopyarea input_leds rfkill sysfillrect pcspkr fb_sys_fops uio_pdrv_genirq led_class i2c_piix4 button uio psmouse mac_hid sch_fq_codel fuse drm dmi_sysfs qemu_fw_cfg ip_tables x_tables ext4 crc32c_generic mbcache crc16 jbd2 virtio_net net_failover virtio_blk failover virtio_balloon ata_generic serio_raw crc32_pclmul crc32c_intel atkbd sha512_ssse3 uhci_hcd pata_acpi vivaldi_fmap aesni_intel ehci_pci libps2 crypto_simd ehci_hcd cryptd i8042 usbcore virtio_pci virtio_pci_legacy_dev floppy ata_piix usb_common serio virtio_pci_modern_dev
[ 1285.506399][  T440] ---[ end trace 0000000000000000 ]---
[ 1285.507383][  T440] RIP: 0010:get_kallsyms_address+0xe6/0x150 [lkrg]
[ 1285.508755][  T440] Code: 28 48 89 05 f4 7c 02 00 48 89 e7 e8 44 e4 81 ec 4c 8b 1d e5 7c 02 00 48 c7 c7 41 06 9a c0 41 ba 91 dc 81 55 45 03 53 fc 74 02 <0f> 0b 41 ff d3 66 90 48 89 05 74 7d 02 00 31 db eb 10 48 c7 c3 ff
[ 1285.511014][  T440] RSP: 0018:ffffb18881bbf998 EFLAGS: 00010202
[ 1285.511740][  T440] RAX: 0000000000000000 RBX: ffffffffc0911005 RCX: 0000000000000000
[ 1285.512602][  T440] RDX: 0000000000384a83 RSI: 0000000000000000 RDI: ffffffffc09a0641
[ 1285.513466][  T440] RBP: ffffb18881bbfd70 R08: ffff97d9847ee0c0 R09: ffff97d980042800
[ 1285.514408][  T440] R10: 0000000055a0ebf7 R11: ffffffffad176a94 R12: ffffffffaea19728
[ 1285.515326][  T440] R13: 0000000000000000 R14: ffff97d9848e02a0 R15: 0000000000000000
[ 1285.516246][  T440] FS:  000066380b197740(0000) GS:ffff97dab7cc0000(0000) knlGS:0000000000000000
[ 1285.517236][  T440] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1285.518069][  T440] CR2: 000066380aaf4400 CR3: 000000010a1da001 CR4: 0000000000760ee0
[ 1285.518984][  T440] PKRU: 55555554
Segmentation fault

on the bright side, ought to mess up some (less friendly) rootkits this way :wink:

[sempervictus]

@Adam-pi3 - suggest trying to build with LLVM16 - it warns quite a bit about:

security/lkrg/modules/exploit_detection/syscalls/compat/p_compat_sys_keyctl/../../../../../modules/exploit_detection/syscalls/p_security_ptrace_access/p_security_ptrace_access.h:21:9: warning: 'P_LKRG_EXPLOIT_DETECTION_SECURITY_PTRACE_ACCESS_H' is used as a header guard here, followed by #define of a different macro [-Wheader-guard]

and tons of:

security/lkrg/modules/exploit_detection/syscalls/compat/p_compat_sys_keyctl/../../../../../modules/exploit_detection/p_exploit_detection.h:498:16: warning: no case matching constant switch condition '0'

[sempervictus]

Built-in, with kCFI, i get (with lkrg.umh_enforce=0 and lkrg.umh_validate=0):

[Adam-pi3]

You are hitting this issue: https://github.com/lkrg-org/lkrg/issues/135 and the solution for that should be the same:

https://github.com/lkrg-org/lkrg/issues/135#issuecomment-1018693257

[sempervictus]

@Adam-pi3 - i have

diff --git a/kernel/seccomp.c b/kernel/seccomp.c
index e9852d1b4a5e..e430748a32b2 100644
--- a/kernel/seccomp.c
+++ b/kernel/seccomp.c
@@ -542,6 +542,8 @@ static void __put_seccomp_filter(struct seccomp_filter *orig)
        }
 }

+EXPORT_SYMBOL(__put_seccomp_filter);
+
 static void __seccomp_filter_release(struct seccomp_filter *orig)
 {
        /* Notify about any unused filters in the task's former filter tree. */

and when built directly into the kernel binary from within the tree, i get this at boot on Arch Linux:

[    1.798112][    T1] CFI failure at get_kallsyms_address+0xfb/0x160 (target: kallsyms_lookup_name+0x4/0x210; expected type: 0xaa7e236f)
[    1.800203][    T1] invalid opcode: 0000 [#1] PREEMPT SMP
[    1.801102][    T1] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G                TN 6.1.9 #1
[    1.802496][    T1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.16.1-1-1 04/01/2014
[    1.804241][    T1] RIP: 0010:get_kallsyms_address+0xfb/0x160
[    1.805224][    T1] Code: bc 24 80 00 00 00 be 01 00 00 00 e8 df 98 a1 ff 4c 8b 1d b0 90 10 01 48 c7 c7 c9 b2 66 bc 41 ba 91 dc 81 55 45 03 53 fc 74 02 <0f> 0b 41 ff d3 66 90 48 89 05 3f 91 10 01 31 db eb 10 48 c7 c3 ff
[    1.808558][    T1] RSP: 0018:ffffa8cbc0013b00 EFLAGS: 00010202
[    1.809580][    T1] RAX: 0000000000000000 RBX: ffffa8cbc0013b00 RCX: 0000000000000000
[    1.810922][    T1] RDX: 0000000000000c82 RSI: 0000000000000000 RDI: ffffffffbc66b2c9
[    1.812312][    T1] RBP: ffffa8cbc0013ee0 R08: ffff8ec1c02a7a80 R09: ffff8ec1c0042800
[    1.813670][    T1] R10: 0000000055a0ebf7 R11: ffffffffbb176cd4 R12: ffffffffbca19728
[    1.815017][    T1] R13: 0000000000000000 R14: ffffffffbd29d988 R15: 0000000000000000
[    1.816432][    T1] FS:  0000000000000000(0000) GS:ffff8ec2f7c80000(0000) knlGS:0000000000000000
[    1.817977][    T1] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.819158][    T1] CR2: 00006775e9ab7798 CR3: 000000002fa0f001 CR4: 0000000000760ee0
[    1.820586][    T1] PKRU: 55555554
[    1.821225][    T1] Call Trace:
[    1.821819][    T1]  <TASK>
[    1.822352][    T1]  ? kallsyms_lookup_name+0x4/0x210
[    1.823298][    T1]  ? __cfi_p_tmp_kprobe_handler+0x10/0x10
[    1.824322][    T1]  ? 0xffffffffc035b000
[    1.825075][    T1]  ? __cfi___initstub__kmod_lkrg__592_655_p_lkrg_register7s+0x5/0x5
[    1.827329][    T1]  p_lkrg_register+0x13/0x6df
[    1.828215][    T1]  ? __cfi___initstub__kmod_lkrg__592_655_p_lkrg_register7s+0x5/0x5
[    1.829667][    T1]  do_one_initcall+0x12d/0x2b0
[    1.830537][    T1]  do_initcall_level+0x72/0x97
[    1.831404][    T1]  do_initcalls+0x46/0x75
[    1.832197][    T1]  kernel_init_freeable+0x127/0x18e
[    1.833141][    T1]  ? __cfi_kernel_init+0x10/0x10
[    1.834061][    T1]  kernel_init+0x15/0x1a0
[    1.834858][    T1]  ret_from_fork+0x1f/0x30
[    1.835678][    T1]  </TASK>
[    1.836227][    T1] Modules linked in:
[    1.836960][    T1] ---[ end trace 0000000000000000 ]---
[    1.837998][    T1] RIP: 0010:get_kallsyms_address+0xfb/0x160
[    1.839171][    T1] Code: bc 24 80 00 00 00 be 01 00 00 00 e8 df 98 a1 ff 4c 8b 1d b0 90 10 01 48 c7 c7 c9 b2 66 bc 41 ba 91 dc 81 55 45 03 53 fc 74 02 <0f> 0b 41 ff d3 66 90 48 89 05 3f 91 10 01 31 db eb 10 48 c7 c3 ff
[    1.842982][    T1] RSP: 0018:ffffa8cbc0013b00 EFLAGS: 00010202
[    1.844122][    T1] RAX: 0000000000000000 RBX: ffffa8cbc0013b00 RCX: 0000000000000000
[    1.845649][    T1] RDX: 0000000000000c82 RSI: 0000000000000000 RDI: ffffffffbc66b2c9
[    1.847180][    T1] RBP: ffffa8cbc0013ee0 R08: ffff8ec1c02a7a80 R09: ffff8ec1c0042800
[    1.848772][    T1] R10: 0000000055a0ebf7 R11: ffffffffbb176cd4 R12: ffffffffbca19728
[    1.850311][    T1] R13: 0000000000000000 R14: ffffffffbd29d988 R15: 0000000000000000
[    1.851840][    T1] FS:  0000000000000000(0000) GS:ffff8ec2f7c80000(0000) knlGS:0000000000000000
[    1.853553][    T1] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.854815][    T1] CR2: 00006775e9ab7798 CR3: 000000002fa0f001 CR4: 0000000000760ee0
[    1.857167][    T1] PKRU: 55555554
[    1.857930][    T1] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    1.859352][    T1] Kernel Offset: 0x3a000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.860904][    T1] Rebooting in 120 seconds..

[Adam-pi3]

Yes, this is kCFI related issue. However, in your previous screen you certainly had a problem with the symbol:

[sempervictus]

Pardon, i think i'm having some problems w/ debian builds for the kernel w/ LLVM - bloody thing has doxygen deps on llvm 11 apparently, and i'm not 100% on what its producing. The Arch ones seem to be fine The kCFI concern however does seem a somewhat serious problem - guessing it needs compiler-visible function calls to the relevant sites to produce acceptable bitmaps?