Closed hedmo closed 3 years ago
Adam-pi3
commented
3 years ago Hi. I believe that the problem is your non-standard kernel configuration where you did enable CONFIG_FTRACE and CONFIG_FUNCTION_TRACER. However, at the same time you did not enable CONFIG_DYNAMIC_FTRACE. I've just pushed the commit where we add this verification as well. Can you try that fix?
hedmo
commented
3 years ago @Adam-pi3 thank you . adding CONFIG_DYNAMIC_FTRACE to my .config made it compile. before we close the issue.. are they mandatory?
solardiz
commented
3 years ago @hedmo My reading of @Adam-pi3's commit is that it adds support for kernels compiled without CONFIG_DYNAMIC_FTRACE, so can you please test that with the latest LKRG code including this commit? Thanks! (I hope Adam will correct me if I am wrong.)
Adam-pi3
commented
3 years ago @solardiz you are correct. https://github.com/openwall/lkrg/commit/b2d193b5ec7cb8aee925c0990e356f0b9c35ffed should correctly handle kernels compiled with and without CONFIG_DYNAMIC_FTRACE if I didn't overlook any corner case.
hedmo
commented
3 years ago yes the latest commit : ( https://github.com/openwall/lkrg/commit/b2d193b5ec7cb8aee925c0990e356f0b9c35ffed) did work by adding DYNAMIC_FTRACE but fails without. but my friend did not manage to make it load : https://defuse.ca/b/shcGsgmw . is there any mandatory kernel options that needs to be added? . as i am running gentoo and maintain the lkrg ebuild i like to add the dependencies in the ebuild to not get unnecessary issues because of wrong kernel setup.
and thanks for the answers
Adam-pi3
commented
3 years ago I've verified that LKRG correctly compiles and runs with and without DYNAMIC_FTRACE feature:
root@oi3:~/lkrg# make clean; make
make -C /lib/modules/5.11.0-rc3-g3-00017-ga0d54b4f5b21/build M=/root/lkrg clean
make[1]: Entering directory '/root/linux'
CLEAN /root/lkrg/Module.symvers
make[1]: Leaving directory '/root/linux'
rm -f Module.markers modules.order
rm -f /root/lkrg/src/modules/kmod/client/kmod/Module.markers
rm -f /root/lkrg/src/modules/kmod/client/kmod/modules.order
rm -f -rf output
make -C /lib/modules/5.11.0-rc3-g3-00017-ga0d54b4f5b21/build M=/root/lkrg modules
make[1]: Entering directory '/root/linux'
CC [M] /root/lkrg/src/modules/ksyms/p_resolve_ksym.o
CC [M] /root/lkrg/src/modules/hashing/p_lkrg_fast_hash.o
CC [M] /root/lkrg/src/modules/comm_channel/p_comm_channel.o
CC [M] /root/lkrg/src/modules/integrity_timer/p_integrity_timer.o
CC [M] /root/lkrg/src/modules/kmod/p_kmod.o
CC [M] /root/lkrg/src/modules/database/CPU.o
CC [M] /root/lkrg/src/modules/database/arch/x86/p_x86_metadata.o
CC [M] /root/lkrg/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.o
CC [M] /root/lkrg/src/modules/database/arch/arm64/p_arm64_metadata.o
CC [M] /root/lkrg/src/modules/database/arch/arm/p_arm_metadata.o
CC [M] /root/lkrg/src/modules/database/arch/p_arch_metadata.o
CC [M] /root/lkrg/src/modules/database/JUMP_LABEL/p_arch_jump_label_transform/p_arch_jump_label_transform.o
CC [M] /root/lkrg/src/modules/database/JUMP_LABEL/p_arch_jump_label_transform_apply/p_arch_jump_label_transform_apply.o
CC [M] /root/lkrg/src/modules/database/FTRACE/p_ftrace_modify_all_code/p_ftrace_modify_all_code.o
CC [M] /root/lkrg/src/modules/database/FTRACE/p_ftrace_enable_sysctl/p_ftrace_enable_sysctl.o
CC [M] /root/lkrg/src/modules/database/p_database.o
CC [M] /root/lkrg/src/modules/notifiers/p_notifiers.o
CC [M] /root/lkrg/src/modules/self-defense/hiding/p_hiding.o
CC [M] /root/lkrg/src/modules/exploit_detection/p_rb_ed_trees/p_rb_ed_pids/p_rb_ed_pids_tree.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_install.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_search_binary_handler/p_search_binary_handler.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_call_usermodehelper/p_call_usermodehelper.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_call_usermodehelper_exec/p_call_usermodehelper_exec.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_do_exit/p_do_exit.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_wake_up_new_task/p_wake_up_new_task.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setuid/p_sys_setuid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setreuid/p_sys_setreuid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setresuid/p_sys_setresuid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setfsuid/p_sys_setfsuid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setgid/p_sys_setgid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setregid/p_sys_setregid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setresgid/p_sys_setresgid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setfsgid/p_sys_setfsgid.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_set_current_groups/p_set_current_groups.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_generic_permission/p_generic_permission.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sel_write_enforce/p_sel_write_enforce.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_seccomp/p_seccomp.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_unshare/p_sys_unshare.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_sys_setns/p_sys_setns.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/caps/p_sys_capset/p_sys_capset.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/caps/p_cap_task_prctl/p_cap_task_prctl.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/keyring/p_key_change_session_keyring/p_key_change_session_keyring.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/keyring/p_sys_add_key/p_sys_add_key.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/keyring/p_sys_request_key/p_sys_request_key.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/keyring/p_sys_keyctl/p_sys_keyctl.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_security_ptrace_access/p_security_ptrace_access.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/compat/p_compat_sys_keyctl/p_compat_sys_keyctl.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/compat/p_compat_sys_capset/p_compat_sys_capset.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/compat/p_compat_sys_add_key/p_compat_sys_add_key.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/compat/p_compat_sys_request_key/p_compat_sys_request_key.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_keyctl/p_x32_sys_keyctl.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/override/p_override_creds/p_override_creds.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/override/p_revert_creds/p_revert_creds.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/override/overlayfs/p_ovl_create_or_link/p_ovl_create_or_link.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/pCFI/p_mark_inode_dirty/p_mark_inode_dirty.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/pCFI/p_schedule/p_schedule.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/pCFI/p___queue_work/p___queue_work.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/pCFI/p_lookup_fast/p_lookup_fast.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_capable/p_capable.o
CC [M] /root/lkrg/src/modules/exploit_detection/syscalls/p_scm_send/p_scm_send.o
CC [M] /root/lkrg/src/modules/exploit_detection/p_exploit_detection.o
CC [M] /root/lkrg/src/p_lkrg_main.o
LD [M] /root/lkrg/p_lkrg.o
MODPOST /root/lkrg/Module.symvers
CC [M] /root/lkrg/p_lkrg.mod.o
LD [M] /root/lkrg/p_lkrg.ko
make[1]: Leaving directory '/root/linux'
mkdir -p output
cp /root/lkrg/p_lkrg.ko output
root@oi3:~/lkrg# insmod output/p_lkrg.ko
root@oi3:~/lkrg# cat /boot/config-`uname -r`|grep CONFIG_DYNAMIC_FTRACE
# CONFIG_DYNAMIC_FTRACE is not set
root@oi3:~/lkrg# dmesg |grep LKRG|grep init
[ 350.478401] [p_lkrg] LKRG initialized successfully!
root@oi3:~/lkrg#The problem which you see during load time is unrelated to FTRACE. Most likely you have optimized SECCOMP functions or don't have that feature. Can you tell me what is an output from the command:
cat /proc/kallsyms |grep put_seccomp_filter
and what is the kernel configuration for SECCOMP e.g. via command:
cat /boot/config-uname -r|grep CONFIG_|grep SECCOMP
Thanks, Adam
0xC0ncord
commented
3 years ago @hedmo Just wanted to drop in here and let you know that I am also running LKRG on Gentoo with the latest stable kernel (5.10.7 right now). I haven't had any issues thus far, and would be glad to help provide my config if you need it.
hedmo
commented
3 years ago hi again . sorry for a late reply. @Adam-pi3 here is the log for SECCOMP : https://defuse.ca/b/KcINeLLh . @0xC0ncord that is good to hear . i have a "WORK IN PROGRESS" ebuild at my git . it is working for me but my friend is running on a RT kernel and have problem to make the LKRG module load.ATM i am on 5.10.8-xanmod-cacule with a working LKRG module :).
Adam-pi3
commented
3 years ago @hedmo it looks like the function __put_seccomp_filter is inlined in your friends environment. Is it possible that he can try to add noinline attribute to this function and verify if my presumptions are correct?
solardiz
commented
3 years ago yes the latest commit : ( b2d193b) did work by adding DYNAMIC_FTRACE but fails without.
BTW, this is still confusing - it's inconsistent with Adam's testing. @hedmo Can you show whether/how current LKRG "fails without" DYNAMIC_FTRACE?
The seccomp issue is a separate one.
hedmo
commented
3 years ago yes the latest commit : ( b2d193b) did work by adding DYNAMIC_FTRACE but fails without.
BTW, this is still confusing - it's inconsistent with Adam's testing. @hedmo Can you show whether/how current LKRG "fails without" DYNAMIC_FTRACE?
The seccomp issue is a separate one.
@solardiz i Will remove DYNAMIC_FTRACE when i update to 5.10.9 and report back . Maybe i forgot to use THE latest commit...
hedmo
commented
3 years ago i can confirm that it compiles without DYNAMIC_FTRACE.and for the SECCOMP problem : https://defuse.ca/b/d3FNUy9c
solardiz
commented
3 years ago @hedmo Thanks. In that pastebin, you have "Can't hook do_execveat_common". This message cannot be produced by latest LKRG, and it's unrelated to seccomp. Please try latest LKRG. Also, please put logs directly into GitHub comments, not via the pastebin service.
osevan
commented
3 years ago Me are the friend
solardiz
commented
3 years ago @osevan Thanks. So the relevant lines are:
Jan 20 23:08:29 ex kernel: [ 303.535392] [p_lkrg] [kretprobe] register_kretprobe() for <do_seccomp> failed! [err=-22]
Jan 20 23:08:29 ex kernel: [ 303.535403] [p_lkrg] Trying to find ISRA / CONSTPROP name for <do_seccomp>
Jan 20 23:08:29 ex kernel: [ 303.632770] [p_lkrg] [kretprobe] register_kretprobe() for do_seccomp failed and ISRA / CONSTPROP version not found!What is your kernel version and config, and compiler version the kernel was built with?
solardiz
commented
3 years ago @Adam-pi3 Currently we're hooking one of these functions:
#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,17,0)
.kp.symbol_name = "do_seccomp",
#else
.kp.symbol_name = "prctl_set_seccomp",
#endifIt looks like because of possible inlining of do_seccomp on 3.17+ we need to hook two functions: prctl_set_seccomp and sys_seccomp. I think the same handler functions can be reused for both hooks. We just need to always install them for prctl_set_seccomp and on 3.17+ also for sys_seccomp.
Edit: I missed that there's also usage of and an issue with __put_seccomp_filter. That's trickier.
osevan
commented
3 years ago @osevan Thanks. So the relevant lines are:
Jan 20 23:08:29 ex kernel: [ 303.535392] [p_lkrg] [kretprobe] register_kretprobe() for <do_seccomp> failed! [err=-22] Jan 20 23:08:29 ex kernel: [ 303.535403] [p_lkrg] Trying to find ISRA / CONSTPROP name for <do_seccomp> Jan 20 23:08:29 ex kernel: [ 303.632770] [p_lkrg] [kretprobe] register_kretprobe() for do_seccomp failed and ISRA / CONSTPROP version not found!What is your kernel version and config, and compiler version the kernel was built with?
RT 5.10.3 with noinline __put_seccomp_filter compiled
Adam-pi3
commented
3 years ago @solardiz I think you are correct. Regardless of that, we should change the logic to update just a seccomp configuration in that code path (not the entire process).
@osevan can you confirm that do_seccomp function is inlined? E.g. in the seccomp syscall and/or inside the function prctl_set_seccomp?
osevan
commented
3 years ago cat /proc/kallsyms |grep -i seccomp 0000000000000000 T arch_seccomp_spec_mitigate 0000000000000000 T audit_seccomp 0000000000000000 T audit_seccomp_actions_logged 0000000000000000 t seccomp_check_filter 0000000000000000 t seccomp_notify_poll 0000000000000000 t seccomp_notify_ioctl 0000000000000000 t seccomp_do_user_notification.constprop.0 0000000000000000 t seccomp_filter_orphan 0000000000000000 t put_seccomp_filter 0000000000000000 t seccomp_notify_release 0000000000000000 t seccomp_actions_logged_handler 0000000000000000 t seccomp_filter 0000000000000000 T seccomp_filter_release 0000000000000000 T get_seccomp_filter 0000000000000000 t seccomp_set_mode_filter 0000000000000000 T x64_sys_seccomp 0000000000000000 T ia32_sys_seccomp 0000000000000000 T prctl_get_seccomp 0000000000000000 T prctl_set_seccomp 0000000000000000 T seccomp_get_filter 0000000000000000 T seccomp_get_metadata 0000000000000000 d seccomp_log_names 0000000000000000 d seccomp_actions_avail 0000000000000000 d seccomp_notify_ops 0000000000000000 d seccomp_sysctl_table 0000000000000000 d seccomp_sysctl_path 0000000000000000 d event_exit__seccomp 0000000000000000 d event_enterseccomp 0000000000000000 d syscall_metaseccomp 0000000000000000 d argsseccomp 0000000000000000 d typesseccomp 0000000000000000 d seccomp_actions_logged 0000000000000000 t seccomp_sysctl_init 0000000000000000 d event_exitseccomp 0000000000000000 d event_enterseccomp 0000000000000000 d p_syscall_meta__seccomp 0000000000000000 d _eiladdria32_sys_seccomp 0000000000000000 d _eil_addr_x64_sys_seccomp 0000000000000000 d initcall_seccomp_sysctl_init6
do_seccomp isnt here need again new compile with noinline before function start?
or we need a solution to handle with inlined functions.
here is my kernel :
https://github.com/xanmod/linux/tree/5.10.3-rt20-xanmod1
only change is in ./kernel/seccomp.c
noinline before __put_seccomp_filter be
i can try with noinline do_seccomp and noinline prctl_set_seccomp, but we need a better solution for future kernel.....
thanks and
best regards
osevan1
commented
3 years ago now i have noinline compile of these three functions and i receive another errors after restart and try to insmod.
but yes after noinline do_seccomp and prctl_set_seccomp error disappear, but new errors come in..... so we need tidy solution for asm inlined kernel compiles.
thanks and
best regards
cmd sudo insmod output/p_lkrg.ko p_init_log_level=3 insmod: ERROR: could not insert module output/p_lkrg.ko: Operation not permitted
Jan 28 21:21:43 ex kernel: [ 187.420794] p_lkrg: loading out-of-tree module taints kernel.
Jan 28 21:21:43 ex kernel: [ 187.421113] p_lkrg: module verification failed: signature and/or required key missing - tainting kernel
Jan 28 21:21:43 ex kernel: [ 187.423669] p_lkrg: unknown parameter 'p_init_log_level' ignored
Jan 28 21:21:43 ex kernel: [ 187.423841] [p_lkrg] Loading LKRG...
Jan 28 21:21:43 ex kernel: [ 187.423844] [p_lkrg] System does NOT support SMEP. LKRG can't enforce SMEP validation :(
Jan 28 21:21:43 ex kernel: [ 187.423845] [p_lkrg] System does NOT support SMAP. LKRG can't enforce SMAP validation :(
Jan 28 21:21:43 ex kernel: [ 187.459616] Freezing user space processes ... (elapsed 0.002 seconds) done.
Jan 28 21:21:43 ex kernel: [ 187.461649] OOM killer disabled.
Jan 28 21:21:43 ex kernel: [ 187.798862] [p_lkrg] [kretprobe] register_kretprobe() for
Adam-pi3
commented
3 years ago Can you also try noinline attribute for change_page_attr_set_clr ? It is informative what else might be inlined in your specific case.
Another question is, what type of aggressive optimization do you use?
osevan
commented
3 years ago Can you also try
noinlineattribute forchange_page_attr_set_clr? It is informative what else might be inlined in your specific case. Another question is, what type of aggressive optimization do you use?
https://github.com/xanmod/linux/tree/5.10.3-rt20-xanmod1 See config file march=native as extra change and new cpu scheduler cachy revision9
osevan
commented
3 years ago @osevan Thanks. So the relevant lines are:
Jan 20 23:08:29 ex kernel: [ 303.535392] [p_lkrg] [kretprobe] register_kretprobe() for <do_seccomp> failed! [err=-22] Jan 20 23:08:29 ex kernel: [ 303.535403] [p_lkrg] Trying to find ISRA / CONSTPROP name for <do_seccomp> Jan 20 23:08:29 ex kernel: [ 303.632770] [p_lkrg] [kretprobe] register_kretprobe() for do_seccomp failed and ISRA / CONSTPROP version not found!What is your kernel version and config, and compiler version the kernel was built with?
Compiler version is GCC 10.2 and binutils 2.35
My config only one addition for mnative=y plus cachy cpu scheduler revision 9
osevan
commented
3 years ago I have now compiled with noinline change_page _attr_set_clr
Errorlog
sudo tail /var/log/syslog
Feb 6 16:07:44 : [ 752.319162] [p_lkrg] Trying to find ISRA / CONSTPROP name for
Kernel 5.10.12-RT26
Adam-pi3
commented
3 years ago I've pushed more changes to LKRG which might address the problem with page attributes (https://github.com/openwall/lkrg/commit/7d45ab04d341ab2235cc364b2a8cfb45302389bb). If you define P_KERNEL_AGGRESSIVE_INLINING (uncomment https://github.com/openwall/lkrg/blob/993be4b6249849abdc33e18d959c29cc6a8aba9e/src/p_lkrg_main.h#L102) LKRG will use different API for page attribute operations. Can you try that?
solardiz
commented
3 years ago Adam, is there any reason not to have the P_KERNEL_AGGRESSIVE_INLINING behavior enabled all the time, removing the pieces of LKRG code that this setting replaces? Maybe once this setting is confirmed to help?
BTW, when you refer to a specific line number on GitHub I suggest you also refer to a specific revision, as otherwise the line number will eventually become wrong. There's the GitHub keyboard shortcut y, which turns the currently opened URL into its equivalent "permalink" (includes the revision in it). I suggest you press this key before copy-pasting this kind of links. Thanks! (And you can edit the comment above now.)
Adam-pi3
commented
3 years ago @solardiz I'm not convinced that we should support by default non-standard aggressively optimized kernel builds (maybe we should?). So far, we haven't seen similar issues anywhere else. There are some side-effects by enabling this logic by default, e.g.:
I believe, it's still an open question which direction we should go. I think we should discuss that offline in more details.
Btw. thanks for the tip about GitHub! Didn't know about that :)
solardiz
commented
3 years ago it's still an open question which direction we should go.
Right. Maybe we should use temporary branches for experiments like this, not to clutter the main codebase with code that we might end up leaving unsupported and possibly broken later.
osevan
commented
3 years ago Feb 11 19:14:31 ex kernel: [ 169.600207] p_lkrg: loading out-of-tree module taints kernel. Feb 11 19:14:31 ex kernel: [ 169.600469] p_lkrg: module verification failed: signature and/or re quired key missing - tainting kernel Feb 11 19:14:31 ex kernel: [ 169.602866] p_lkrg: unknown parameter 'p_init_log_level' ignored Feb 11 19:14:31 ex kernel: [ 169.603018] [p_lkrg] Loading LKRG... Feb 11 19:14:31 ex kernel: [ 169.603020] [p_lkrg] System does NOT support SMEP. LKRG can't enfor ce SMEP validation :( Feb 11 19:14:31 ex kernel: [ 169.603021] [p_lkrg] System does NOT support SMAP. LKRG can't enfor ce SMAP validation :( Feb 11 19:14:32 ex kernel: [ 169.636747] Freezing user space processes ... (elapsed 0.001 second s) done. Feb 11 19:14:32 ex kernel: [ 169.638672] OOM killer disabled. Feb 11 19:14:32 ex kernel: [ 170.027350] [p_lkrg] [kretprobe] register_kretprobe() for <ovl_crea te_or_link> failed! [err=-22] Feb 11 19:14:32 ex kernel: [ 170.027359] [plkrg] Trying to find ISRA / CONSTPROP name for <ovl create_or_link> Feb 11 19:14:32 ex kernel: [ 170.140859] [p_lkrg] [kretprobe] register_kretprobe() for ovl_creat e_or_link failed and ISRA / CONSTPROP version not found! Feb 11 19:14:32 ex kernel: [ 170.140866] [p_lkrg] Can't hook 'ovl_create_or_link' function. This is expected if you are not using OverlayFS. Feb 11 19:14:32 ex kernel: [ 170.210597] [p_lkrg] [kretprobe] register_kretprobe() for <lookup_f ast> failed! [err=-22] Feb 11 19:14:32 ex kernel: [ 170.210607] [p_lkrg] Trying to find ISRA / CONSTPROP name for <look up_fast> Feb 11 19:14:32 ex kernel: [ 170.308428] [p_lkrg] [kretprobe] register_kretprobe() for lookup_fast failed and ISRA / CONSTPROP version not found! Feb 11 19:14:32 ex kernel: [ 170.308435] [p_lkrg] LKRG won't enforce pCFI validation on 'lookup_fast' Feb 11 19:14:32 ex kernel: [ 170.746738] [p_lkrg] LKRG initialized successfully! Feb 11 19:14:32 ex kernel: [ 170.746746] OOM killer enabled.
osevan
commented
3 years ago i loaded like in readme here sudo insmod output/p_lkrg.ko kint_enforce=1
no errors at the moment...
could you give me best insmod method for lkrg.ko ?
osevan
commented
3 years ago Im receiving second freeze and need to.reset with 8 second.hold.power button
:-(
Insmod with kint_enforce=1
Im opening libreoffice 7 times and closing every libreoffice windows.
Starting libreoffice its freeze suddenly with no error outputs
sudo tail -1200 /var/log/syslog |grep "LKRG initialized" -A10 Feb 11 19:45:09 ex kernel: [ 222.250860] [p_lkrg] LKRG initialized successfully! Feb 11 19:45:09 ex kernel: [ 222.250869] OOM killer enabled. Feb 11 19:45:09 ex console-kit-daemon[3008]: WARNING: Error waiting for native console 50 activation: Resource temporarily unavailable Feb 11 19:45:09 ex kernel: [ 222.250870] Restarting tasks ... done. Feb 11 19:45:10 ex ntpd[2298]: error resolving pool 1.debian.pool.ntp.org: Name or service not known (-2) Feb 11 19:45:11 ex ntpd[2298]: error resolving pool 2.debian.pool.ntp.org: Name or service not known (-2) Feb 11 19:45:13 ex ntpd[2298]: error resolving pool 3.debian.pool.ntp.org: Name or service not known (-2) Feb 11 19:48:32 ex liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2122" x-info="http://www.rsyslog.com"] start Feb 11 19:48:32 ex kernel: [ 0.000000] Linux version 5.10.3-rt20-xanmod1c9no (root@ex) (gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35.1) #1 SMP PREEM
Now.freezes 5 times in row always suddenly. :-((((((
Adam-pi3
commented
3 years ago OK, so we solved the problem with loading, thanks for the information :) After loading LKRG, can you change the enforcement profile to 0? You can do it by executing the following command:
# sysctl lkrg.profile_enforce=0
lkrg.profile_enforce = 0
# dmesg|grep profile_enforce
[438944.809518] [p_lkrg] Changing "profile_enforce" logic. From Old[2 | Strict] to new[0 | Log & Accept] one.After running this command, can you try to repeat what you've done which resulted in freezing / crashing and check what is in the logs? Is there anything which LKRG reports?
osevan
commented
3 years ago Ok.thank you I.will.test.tomorrow
osevan
commented
3 years ago Feb 12 12:20:19 ex kernel: [ 1419.628220] [p_lkrg] [kretprobe] register_kretprobe() for
you can see no error logs output only freeze suddenly sometimes 30 secs after sometimes 5 minutes after loaded module....
thanks and
best regards
osevan
commented
3 years ago OK, so we solved the problem with loading, thanks for the information :) After loading LKRG, can you change the enforcement profile to 0? You can do it by executing the following command:
# sysctl lkrg.profile_enforce=0 lkrg.profile_enforce = 0 # dmesg|grep profile_enforce [438944.809518] [p_lkrg] Changing "profile_enforce" logic. From Old[2 | Strict] to new[0 | Log & Accept] one.After running this command, can you try to repeat what you've done which resulted in freezing / crashing and check what is in the logs? Is there anything which LKRG reports?
It comes suddenly but two times in row when I try to login in firefox's with autologin and hit enter.
Firefox is running in firejailed sandbox with x-server sandbox (xpra) as non root Unix sockets
osevan
commented
3 years ago OK, so we solved the problem with loading, thanks for the information :) After loading LKRG, can you change the enforcement profile to 0? You can do it by executing the following command:
# sysctl lkrg.profile_enforce=0 lkrg.profile_enforce = 0 # dmesg|grep profile_enforce [438944.809518] [p_lkrg] Changing "profile_enforce" logic. From Old[2 | Strict] to new[0 | Log & Accept] one.After running this command, can you try to repeat what you've done which resulted in freezing / crashing and check what is in the logs? Is there anything which LKRG reports?
It comes suddenly but two times in row when I try to login in firefox's witj autologin and hit enter.
Firefox is running in firejailed sandbox with x-server sandbox (xpra) as non root Unix sockets
But it comes outside of.Firefox too for example in libreoffice.
Dunnno why it comes suddenly and randomly.
We need better Tracing possibility in Lkrg to.figure out what is the mistake
Adam-pi3
commented
3 years ago Thanks for the test. Most likely you should disable kint_integrity and verify if it helps. You can do it via:
# sysctl lkrg.kint_validate=0
# dmesg |grep kint_validate
[509698.955268] [p_lkrg] Changing "kint_validate" logic. From Old[3 | PERIODICALLY + RANDOM EVENTS] to new[0 | DISABLED] one.We need better Tracing possibility in Lkrg to.figure out what is the mistake
LKRG has a very detailed level of tracing. It is controlled via log_level parameter. Per our documentation:
- lkrg.log_level (3)
LKRG's logging verbosity level. Allowed values are from 0 to 4 for normal
builds or from 0 to 6 for debugging builds.
Values of 4 and higher are meant for debugging only and produce too verbose
logging for production use. Moreover, some messages logged at those high
levels contain information useful for kernel vulnerability exploitation,
making those log levels potentially mildly insecure (depending on other
system configuration).I would suggest to disable kint_validate and verify if that helps. If it does, increase log_level up to the level 5 (5 and 6 is available on debug build) and reenable kint_validate
osevan
commented
3 years ago How I can build with debug options?
Adam-pi3
commented
3 years ago Uncomment the following lines in Makefile:
-# ccflags-m := -ggdb -DP_LKRG_DEBUG_BUILD -finstrument-functions
-# ccflags-y := ${ccflags-m}
-# p_lkrg-objs += src/modules/print_log/p_lkrg_debug_log.o
+ccflags-m := -ggdb -DP_LKRG_DEBUG_BUILD -finstrument-functions
+ccflags-y := ${ccflags-m}
+p_lkrg-objs += src/modules/print_log/p_lkrg_debug_log.o@osevan is it possible for me to set-up the same environment so I would be able to repro your issues? It will significantly help me with debugging.
osevan
commented
3 years ago ofcourse
wget https://github.com/xanmod/linux/archive/5.10.3-rt20-xanmod1.zip dtrx *.zip cd linux-5.10.3-rt20-xanmod1 wget https://raw.githubusercontent.com/hamadmarri/cacule-cpu-scheduler/master/patches/Cachy/v5.9/xanmod-rt/cachy-5.9-r9-rt.patch patch -p1 < cachy-5.9-r9-rt.patch
replace .config with my config attention replace config like with this one else cachy wont compile here is config file. https://defuse.ca/b/Xp9EGx6A
autogroup_sched and fairgroup is disabled because cachy need it
no inlining of these functions in ./kernel/seccomp.c
noinline static void put_seccomp_filter(struct seccomp_filter *orig) noinline static long do_seccomp(unsigned int op, unsigned int flags, void user uargs) noinline long prctl_set_seccomp(unsigned long seccomp_mode, void __user filter) #############################################################################################
osevan
commented
3 years ago Feb 12 21:06:12 ex kernel: [ 1765.913254] [p_lkrg] Leaving function
this time i did nothing only waited after loading module and setting manually sysctl configs
after 3-6 mins freeze again
Feb 12 21:06:15 ex kernel: [ 1769.142159] [p_lkrg] Leaving function
loglevel 6
freezes again with
sudo sysctl lkrg.log_level=6
sudo sysctl lkrg.profile_enforce=0 sudo sysctl lkrg.kint_validate=0
Adam-pi3
commented
3 years ago I've compiled and installed the same kernel on 2 different VMs and I believe I found what's the problem. It looks unrelated to LKRG but under resource pressure LKRG makes that problem more visible. During normal run on Ubuntu 20.04 server version I haven't seen any problems with LKRG and your kernel. When I did the same test on CentOS I also didn't see any bad results. However, when I enabled X window and started to install libreoffice, run firefox at the background etc. I saw in the logs e.g., following entry:
[Mon Feb 15 02:13:49 2021] BUG: scheduling while atomic: gvfs-udisks2-vo/3346/0x00000002
[Mon Feb 15 02:13:49 2021] Modules linked in: nls_utf8 isofs p_lkrg(OE) xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nft_counter msr xt_tcpudp nf_nat_tftp nft_objref nf_conntrack_tftp bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6_tables nft_compat ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr joydev vmw_balloon intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_midi crypto_simd snd_seq_midi_event cryptd glue_helper rapl input_leds psmouse pcspkr snd_ens1371 snd_ac97_codec gameport snd_rawmidi ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 vmw_vmci mac_hid sch_fq_codel ip_tables x_tables xfs libcrc32c vmwgfx drm_kms_helper syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops cec rc_core mptspi ttm
[Mon Feb 15 02:13:49 2021] scsi_transport_spi mptscsih pata_acpi drm e1000 mptbase hid_generic usbmouse usbhid hid autofs4
[Mon Feb 15 02:13:49 2021] CPU: 1 PID: 3346 Comm: gvfs-udisks2-vo Tainted: G OE 5.10.3-rt20-xanmod1-RT #2
[Mon Feb 15 02:13:49 2021] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
[Mon Feb 15 02:13:49 2021] Call Trace:
[Mon Feb 15 02:13:49 2021] dump_stack+0x6d/0x88
[Mon Feb 15 02:13:49 2021] __schedule_bug.cold.100+0x4c/0x58
[Mon Feb 15 02:13:49 2021] __schedule+0x5b8/0x870
[Mon Feb 15 02:13:49 2021] preempt_schedule_lock+0x16/0x40
[Mon Feb 15 02:13:49 2021] rt_write_lock+0x186/0x1a0
[Mon Feb 15 02:13:49 2021] p_cap_task_prctl_entry+0x47/0xc0 [p_lkrg]
[Mon Feb 15 02:13:49 2021] pre_handler_kretprobe+0xa3/0x190
[Mon Feb 15 02:13:49 2021] ? cap_task_prctl+0x1/0x2c0
[Mon Feb 15 02:13:49 2021] ? cap_task_prctl+0x5/0x2c0
[Mon Feb 15 02:13:49 2021] kprobe_ftrace_handler+0x81/0xf0
[Mon Feb 15 02:13:49 2021] ftrace_ops_assist_func+0x93/0x190
[Mon Feb 15 02:13:49 2021] ? preempt_schedule_notrace+0x4f/0xa0
[Mon Feb 15 02:13:49 2021] ? preempt_schedule_notrace_thunk+0x16/0x31
[Mon Feb 15 02:13:49 2021] 0xffffffffc07c30c8
[Mon Feb 15 02:13:49 2021] ? cap_vm_enough_memory+0x80/0x80
[Mon Feb 15 02:13:49 2021] ? cap_task_prctl+0x1/0x2c0
[Mon Feb 15 02:13:49 2021] cap_task_prctl+0x5/0x2c0
[Mon Feb 15 02:13:49 2021] security_task_prctl+0x4e/0x80
[Mon Feb 15 02:13:49 2021] __x64_sys_prctl+0x53/0x610
[Mon Feb 15 02:13:49 2021] do_syscall_64+0x33/0x80
[Mon Feb 15 02:13:49 2021] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Mon Feb 15 02:13:49 2021] RIP: 0033:0x7ff67e7ccaae
[Mon Feb 15 02:13:49 2021] Code: 48 8b 0d dd f3 2b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 9d 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d aa f3 2b 00 f7 d8 64 89 01 48
[Mon Feb 15 02:13:49 2021] RSP: 002b:00007fff11c626e8 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
[Mon Feb 15 02:13:49 2021] RAX: ffffffffffffffda RBX: 00007ff67d9da042 RCX: 00007ff67e7ccaae
[Mon Feb 15 02:13:49 2021] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[Mon Feb 15 02:13:49 2021] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000003
[Mon Feb 15 02:13:49 2021] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ff67dbec184
[Mon Feb 15 02:13:49 2021] R13: 0000000000000000 R14: 0000000000000000 R15: 0000562db78b6ee0That could be problematic:
[Mon Feb 15 02:13:49 2021] BUG: scheduling while atomic: gvfs-udisks2-vo/3346/0x00000002and it shouldn't happening. LKRG is using R/W locks under disabled IRQs for critical paths and reported problem is concerning. I thought maybe I overlooked something in the LKRG logic but it looks good. However, in the logs I can see that entry:
[Mon Feb 15 02:13:49 2021] rt_write_lock+0x186/0x1a0
[Mon Feb 15 02:13:49 2021] p_cap_task_prctl_entry+0x47/0xc0 [p_lkrg]At that specific path LKRG uses write_lock_irqsave and I've never seen rt_write_lock version of this lock. When I looked at the vanilla Linux kernel sources this type of lock does NOT exist (that's why I never seen it):
https://elixir.bootlin.com/linux/latest/A/ident/rt_write_lock
So this is a custom implementation of R/W locks. At the sources which you've shared I've found the following declaration in the file ./include/linux/rwlock_rt.h:
#define write_lock_irqsave(lock, flags) \
do { \
typecheck(unsigned long, flags); \
rt_write_lock(lock); \
flags = 0; \
} while (0)and at the same file you can find:
#define write_lock(lock) rt_write_lock(lock)Essentially, the same logic is execute for write_lock and write_lock_irqsave. In the normal kernel, 2nd version disables IRQs and don't sleep / call schedule(). It is safe to call it from IRQ disabled contexts (which is the case for KPROBE interface). However, in your specific kernel the same function is executed for write_lock and write_lock_irqsave. By itself is not wrong, but looking at the implementation of that function I can see this (./kernel/locking/rwlock-rt.c):
void __lockfunc rt_write_lock(rwlock_t *rwlock)
{
rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
__write_rt_lock(rwlock);
rcu_read_lock();
migrate_disable();
}
EXPORT_SYMBOL(rt_write_lock);where:
static void __write_rt_lock(struct rt_rw_lock *lock)
{
struct rt_mutex *m = &lock->rtmutex;
...
/* Take the rtmutex as a first step */
__rt_spin_lock(m);
...
for (;;) {
...
raw_spin_unlock_irqrestore(&m->wait_lock, flags);
if (atomic_read(&lock->readers) != 0)
preempt_schedule_lock();
raw_spin_lock_irqsave(&m->wait_lock, flags);
...And function preempt_schedule_lock is implemented in ./kernel/sched/core.c:
#ifdef CONFIG_PREEMPT_RT
void __sched notrace preempt_schedule_lock(void)
{
do {
preempt_disable();
__schedule(true, true);
sched_preempt_enable_no_resched();
} while (need_resched());
}Essentially, in your kernel, write_lock_irqsave is not irqsave. If you call it from the context where IRQs are disabled, it can call __schedule() and that's the bug. Most likely that's the root-cause of your problems.
@solardiz I'm not sure what to do with this "bug". It might be worth to keep it somehow visible for the reference (this analyze). However, it's not LKRG's bug here.
solardiz
commented
3 years ago @Adam-pi3 I'm not an expert on Linux kernel locking conventions and I didn't look into your specific findings in detail now, but here are some general observations and thoughts:
You appear to read _irqsave as implying "IRQ safe". I think what this naming actually means is merely saving/restoring of IRQ disabled status, so that a subsequent _irqrestore wouldn't wrongly enable IRQs if they were already disabled by the time of its corresponding _irqsave. This matters e.g. when several of these are nested (for different locks) and the locking primitive itself disables IRQs as part of its implementation for the given architecture. In that case, the IRQs are only re-enabled by the outer _irqrestore. This naming in itself does not guarantee that the primitives are safe to use in IRQ context. The apparent similarity between the words "save" and "safe", and actual relevance of the saving to IRQ context safety, might be confusing here. Relevance to safety doesn't mean they are sufficient for safety.
Documentation/locking/locktypes.rst documents "the rules for use under PREEMPT_RT". I think documented behavior isn't a bug. You might want to search this file for "PREEMPT_RT" (there are many mentions) and see if you can easily make LKRG follow those rules. If not, then:
I suggest we document that LKRG is incompatible with RT kernels.
Makes sense?
Adam-pi3
commented
3 years ago You appear to read
_irqsaveas implying "IRQ safe". I think what this naming actually means is merely saving/restoring of IRQ disabled status, so that a subsequent_irqrestorewouldn't wrongly enable IRQs if they were already disabled by the time of its corresponding_irqsave. This matters e.g. when several of these are nested (for different locks) and the locking primitive itself disables IRQs as part of its implementation for the given architecture. In that case, the IRQs are only re-enabled by the outer_irqrestore. This naming in itself does not guarantee that the primitives are safe to use in IRQ context. The apparent similarity between the words "save" and "safe", and actual relevance of the saving to IRQ context safety, might be confusing here. Relevance to safety doesn't mean they are sufficient for safety.
By "irqsave" I was referencing to the logic which is documented in R/W locks:
"Spinning locks implicitly disable preemption and the lock
...
_irqsave/restore() - Save and disable / restore interrupt disabled state"However, in the logic described above they do not do it. I guess RT kernel has different presumptions.
Documentation/locking/locktypes.rstdocuments "the rules for use under PREEMPT_RT". I think documented behavior isn't a bug. You might want to search this file for "PREEMPT_RT" (there are many mentions) and see if you can easily make LKRG follow those rules.
Thanks for the reference. I went through it and it seems like it documents that RT locks are different:
"The semantics of spinlock_t change with the state of PREEMPT_RT.
...
On a PREEMPT_RT kernel spinlock_t is mapped to a separate implementation
based on rt_mutex which changes the semantics:
- Preemption is not disabled.
- The hard interrupt related suffixes for spin_lock / spin_unlock
operations (_irq, _irqsave / _irqrestore) do not affect the CPU's
interrupt disabled state.
...Essentially, it documents that write_lock_irqsave is not doing irqsave. However, they do suggest that it is still possible to achieve what we need by using raw_spinlock_t:
raw_spinlock_t is a strict spinning lock implementation in all kernels,
including PREEMPT_RT kernels. Use raw_spinlock_t only in real critical
core code, low-level interrupt handling and places where disabling
preemption or interrupts is required, for example, to safely access
hardware state. raw_spinlock_t can sometimes also be used when the
critical section is tiny, thus avoiding RT-mutex overhead.If not, then:
I suggest we document that LKRG is incompatible with RT kernels.
Makes sense?
In theory, LKRG could be adopted to the RT kernels (using raw_spinlock_t). However, I don't have time for that now and I'm not sure we should do it at this stage. I can add the code verifying if CONFIG_RT_MUTEXES or PREEMPT_RT is enabled. In case any of them are enabled, fail to compile LKRG. @solardiz does that sounds good for you?
osevan
commented
3 years ago You appear to read
_irqsaveas implying "IRQ safe". I think what this naming actually means is merely saving/restoring of IRQ disabled status, so that a subsequent_irqrestorewouldn't wrongly enable IRQs if they were already disabled by the time of its corresponding_irqsave. This matters e.g. when several of these are nested (for different locks) and the locking primitive itself disables IRQs as part of its implementation for the given architecture. In that case, the IRQs are only re-enabled by the outer_irqrestore. This naming in itself does not guarantee that the primitives are safe to use in IRQ context. The apparent similarity between the words "save" and "safe", and actual relevance of the saving to IRQ context safety, might be confusing here. Relevance to safety doesn't mean they are sufficient for safety.By "irqsave" I was referencing to the logic which is documented in R/W locks:
"Spinning locks implicitly disable preemption and the lock ... _irqsave/restore() - Save and disable / restore interrupt disabled state"However, in the logic described above they do not do it. I guess RT kernel has different presumptions.
Documentation/locking/locktypes.rstdocuments "the rules for use under PREEMPT_RT". I think documented behavior isn't a bug. You might want to search this file for "PREEMPT_RT" (there are many mentions) and see if you can easily make LKRG follow those rules.Thanks for the reference. I went through it and it seems like it documents that RT locks are different:
"The semantics of spinlock_t change with the state of PREEMPT_RT. ... On a PREEMPT_RT kernel spinlock_t is mapped to a separate implementation based on rt_mutex which changes the semantics: - Preemption is not disabled. - The hard interrupt related suffixes for spin_lock / spin_unlock operations (_irq, _irqsave / _irqrestore) do not affect the CPU's interrupt disabled state. ...Essentially, it documents that
write_lock_irqsaveis not doingirqsave. However, they do suggest that it is still possible to achieve what we need by usingraw_spinlock_t:raw_spinlock_t is a strict spinning lock implementation in all kernels, including PREEMPT_RT kernels. Use raw_spinlock_t only in real critical core code, low-level interrupt handling and places where disabling preemption or interrupts is required, for example, to safely access hardware state. raw_spinlock_t can sometimes also be used when the critical section is tiny, thus avoiding RT-mutex overhead.If not, then: I suggest we document that LKRG is incompatible with RT kernels. Makes sense?
In theory, LKRG could be adopted to the RT kernels (using
raw_spinlock_t). However, I don't have time for that now and I'm not sure we should do it at this stage. I can add the code verifying if CONFIG_RT_MUTEXES or PREEMPT_RT is enabled. In case any of them are enabled, fail to compile LKRG. @solardiz does that sounds good for you?
Thank you very much for finding the exact issue.
Some important notes about rt kernels:
Rt kernels are used in time critical area like medicine robots and lazer machines - its important for everyone, who need operations by doctors lazer robots for eyes or for lifesaving products in hospitals.
So we need everytime rt kernels if the mission is critical and in life threatening situations.
If you could fix this for rt kernels, every robotic product in medicine will benefit of Lkrg and may save lives.
Think twice, when it comes to life and enrichment of medicine robots and vital products made
https://www.embedded.com/real-time-kernels-in-medical-device-development-esc-214/
Thanks and
Best regards
osevan
commented
3 years ago 
solardiz
commented
3 years ago @osevan I think you've just given us an excellent reason to refrain from supporting RT kernels. For medical and similar devices, reliability is crucial. LKRG doesn't improve reliability - at best, it doesn't hurt it, although it might. LKRG aims to reduce the number of security compromises across a "population" of Linux systems and over time - yet for individual systems and occasionally it might end up actually hurting security as well. This trading of reliability for probabilistic/statistical improvement in security is, I think, currently unsuitable for applications involving life-safety from the points of view of (1) distribution of life-threatening incidents seen so far (I think by far most were unintentional, not malicious), (2) ethical (do you dare to probabilistically kill someone even if this might save someone else?), and (3) potentially legal (although indeed there are disclaimers in free software licenses, etc.) Of course, such non-trivial trade-offs happen all the time in medicine, and yes people are being probabilistically killed by risky treatment (or even examination) that has an expected net positive balance of benefit vs. risk, but it's not something we're in a position to get into.
So really, let's not support RT kernels in LKRG and be done with this.
osevan
commented
3 years ago :-(
solardiz
commented
3 years ago @osevan No regrets. Sound reasoning is important, and I think we've arrived at it here, in part due to you. Thank you!
hi i was trying to check lkrg out but it fails on me. i dont have much to go on but i give you the buildlog.
regards build.log