search

bytedance / Elkeid

Elkeid is an open source solution that can meet the security requirements of various workloads such as hosts, containers and K8s, and serverless. It is derived from ByteDance's internal best practices.
https://elkeid.bytedance.com
2.24k stars 432 forks source link

重启agent，未执行插入hids_drive模块操作 #251

Closed 0xlwoe21k closed 2 years ago

0xlwoe21k commented 2 years ago

如题：

[root@aliyun-10-43-28-28 driver]# service elkeid-agent restart  
[root@aliyun-10-43-28-28 driver]# lsmod |grep hids 
[root@aliyun-10-43-28-28 driver]# ls 
driver  driver.log  driver.stderr  hids_driver_latest_2.6.32-696.30.1.el6.x86_64_amd64.ko
[root@aliyun-10-43-28-28 driver]# insmod hids_driver_latest_2.6.32-696.30.1.el6.x86_64_amd64.ko 
[root@aliyun-10-43-28-28 driver]# lsmod | grep hids
hids_driver           137949  0
[root@aliyun-10-43-28-28 driver]#

且日志中有如下错误信息：

[root@aliyun-10-43-28-28 driver]# cat driver.log 
2022-06-07T15:33:58.609645659+08:00     ERROR   driver  src/main.rs:35  when loading kernel module,an error occurred: load module failed: ENOSYS: Function not implemented

单独运行driver插件程序，有如下日志信息：

2022-06-07T15:41:07.181329755+08:00     INFO    driver  src/main.rs:39  init kmod successfully
2022-06-07T15:41:07.181422550+08:00     INFO    driver  src/main.rs:57  task receive handler is running
2022-06-07T15:41:07.181500940+08:00     INFO    driver  src/main.rs:63  init ringbuf successfully
2022-06-07T15:41:07.181520623+08:00     INFO    driver::kmod    src/kmod.rs:194 heartbeat: {"filtered_exe_entries": "[]", "udp_recvmsg_kprobe_state": "0", "udpv6_recvmsg_kprobe_state": "0", "filtered_argv_entries": "[]"}
2022-06-07T15:41:09.957363648+08:00     ERROR   driver  src/main.rs:51  when receiving task,an error occurred:unexpected wire type
2022-06-07T15:41:10.273722374+08:00     INFO    driver  src/main.rs:98  plugin will exit
Percivalll commented 2 years ago

execute insmod and than execute dmesg print the log here

0xlwoe21k commented 2 years ago

execute insmod and than execute dmesg print the log here

[root@aliyun-10-43-28-27 driver]# dmesg | tail -n 20
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
type=1305 audit(1612493054.472:3): audit_pid=1188 old=0 auid=4294967295 ses=4294967295 res=1
Clocksource tsc unstable (delta = 757874305 ns)
hids_driver: create 41 print event class
[ELKEID] Filter Init Success 
[ELKEID] compat_sys_execve register_kprobe failed, returned -22
[ELKEID] do_init_module register_kprobe failed, returned -22
[ELKEID] SANDBOX: 0
[ELKEID] register_kprobe success: connect_hook: 1,load_module_hook: 1,execve_hook: 1,call_usermodehekoer_hook: 1,bind_hook: 1,create_file_hook: 1,file_permission_hook: 0, ptrace_hook: 1, update_cred_hook: 1, dns_hook: 1, accept_hook:0, mprotect_hook: 0, mount_hook: 1, link_hook: 1, memfd_create: 1, rename_hook: 1,setsid_hook:1, prctl_hook:1, open_hook:0, udev_notifier:1, nanosleep_hook:0, kill_hook: 0, rm_hook: 0,  exit_hook: 0, write_hook: 0, EXIT_PROTECT: 0
[ELKEID] ANTI_ROOTKIT_CHECK: 1
[ELKEID] uninstall_kprobe success
hids_driver: destroy 41 print event class
hids_driver: create 41 print event class
[ELKEID] Filter Init Success 
[ELKEID] compat_sys_execve register_kprobe failed, returned -22
[ELKEID] do_init_module register_kprobe failed, returned -22
[ELKEID] SANDBOX: 0
[ELKEID] register_kprobe success: connect_hook: 1,load_module_hook: 1,execve_hook: 1,call_usermodehekoer_hook: 1,bind_hook: 1,create_file_hook: 1,file_permission_hook: 0, ptrace_hook: 1, update_cred_hook: 1, dns_hook: 1, accept_hook:0, mprotect_hook: 0, mount_hook: 1, link_hook: 1, memfd_create: 1, rename_hook: 1,setsid_hook:1, prctl_hook:1, open_hook:0, udev_notifier:1, nanosleep_hook:0, kill_hook: 0, rm_hook: 0,  exit_hook: 0, write_hook: 0, EXIT_PROTECT: 0
[ELKEID] ANTI_ROOTKIT_CHECK: 1
Percivalll commented 2 years ago

插件没办法单独运行，所以最后一个日志是正常的

Percivalll commented 2 years ago

重启agent 等待完成之后 不要做任何操作 如果driver进程没退出，等待5min把完整driver日志发一下 如果期间退出了，也发一下完整driver日志

0xlwoe21k commented 2 years ago

agent重启后，driver没起来

补充一个信息，我的内核版本：2.6.32-696.30.1.el6.x86_64

agent的日志：

2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:71 ++++++++++++++++++++++++++++++startup++++++++++++++++++++++++++++++
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:72 product:elkeid-agent
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:73 version:1.7.0.1
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:74 id:3bba2bab-b1fa-564d-9e82-4831611a234b
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:75 hostname:{aliyun-10-43-28-27.ecom-mgproc-prod.intra.openredcloud.com}
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:76 intranet_ipv4:{[10.43.28.27]}
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:77 intranet_ipv6:{[]}
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:78 extranet_ipv4:{[]}
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:79 extranet_ipv6:{[]}
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:80 platform:centos
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:81 platform_family:rhel
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:82 platform_version:6.9
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:83 kernel_version:2.6.32-696.30.1.el6.x86_64
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:84 arch:x86_64
2022-06-07T17:21:35.340+0800    INFO    agent-source/main.go:87 ++++++++++++++++++++++++++++++running++++++++++++++++++++++++++++++
2022-06-07T17:21:35.340+0800    INFO    heartbeat/heartbeat.go:139      health daemon startup
2022-06-07T17:21:35.340+0800    INFO    plugin/plugin.go:176    plugin daemon startup
2022-06-07T17:21:35.340+0800    INFO    transport/transport.go:13       transport daemon startup
2022-06-07T17:21:35.343+0800    INFO    heartbeat/heartbeat.go:98       agent heartbeat completed:map[arch:x86_64 boot_at:1612493045 cpu:0.00000000 du:27534634 fd_cnt:8 grs:12 idc:default kernel_version:2.6.32-696.30.1.el6.x86_64 load_1:0.11 load_15:0.01 load_5:0.06 net_mode:unknown nproc:2 pid:15836 platform:centos platform_family:rhel platform_version:6.9 read_speed:0.00000000 region:default rss:7249920 running_procs:5 rx_speed:0.00000000 rx_tps:0.00000000 started_at:1654593694 total_procs:365 tx_speed:0.00000000 tx_tps:0.00000000 write_speed:20480.00000000]
2022-06-07T17:21:41.364+0800    INFO    transport/transfer.go:59        get connection successfully:idc default,region default,netmode private
2022-06-07T17:21:41.365+0800    INFO    transport/transfer.go:149       receive handler running
2022-06-07T17:21:41.365+0800    INFO    transport/transfer.go:91        send handler running
2022-06-07T17:21:41.470+0800    INFO    transport/transfer.go:156       received command
2022-06-07T17:21:41.470+0800    INFO    plugin/plugin.go:197    syncing plugins...
2022-06-07T17:21:41.470+0800    INFO    plugin/plugin_linux.go:61       plugin is loading...    {"plugin": "collector", "pver": "1.0.0.81", "psign": "2e3225f66bb267411cc82b13c982656fdabde10e3d55161f0a6d999594fc0d16"}
2022-06-07T17:21:41.585+0800    INFO    plugin/plugin_linux.go:100      plugin's process will start     {"plugin": "collector", "pver": "1.0.0.81", "psign": "2e3225f66bb267411cc82b13c982656fdabde10e3d55161f0a6d999594fc0d16"}
2022-06-07T17:21:41.585+0800    INFO    plugin/plugin.go:209    plugin has been loaded  {"plugin": "collector", "pver": "1.0.0.81", "psign": "2e3225f66bb267411cc82b13c982656fdabde10e3d55161f0a6d999594fc0d16"}
2022-06-07T17:21:41.586+0800    INFO    plugin/plugin_linux.go:61       plugin is loading...    {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.879+0800    INFO    plugin/plugin_linux.go:100      plugin's process will start     {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.879+0800    INFO    plugin/plugin.go:209    plugin has been loaded  {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.879+0800    INFO    plugin/plugin_linux.go:61       plugin is loading...    {"plugin": "journal_watcher", "pver": "1.0.0.10", "psign": "982d575e604f2118622faa50099759d5c3b021ec549f0440f5703e4a8968709d"}
2022-06-07T17:21:41.884+0800    INFO    plugin/plugin_linux.go:100      plugin's process will start     {"plugin": "journal_watcher", "pver": "1.0.0.10", "psign": "982d575e604f2118622faa50099759d5c3b021ec549f0440f5703e4a8968709d"}
2022-06-07T17:21:41.978+0800    INFO    plugin/plugin.go:209    plugin has been loaded  {"plugin": "journal_watcher", "pver": "1.0.0.10", "psign": "982d575e604f2118622faa50099759d5c3b021ec549f0440f5703e4a8968709d"}
2022-06-07T17:21:41.978+0800    INFO    plugin/plugin_linux.go:61       plugin is loading...    {"plugin": "scanner", "pver": "3.0.0.5", "psign": "a59f4a2f9f2ea582e787389536b18f33c340057797764e215f94ea2861d54ada"}
2022-06-07T17:21:41.980+0800    INFO    plugin/plugin_linux.go:151      gorountine of receiving plugin's data will exit {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.980+0800    INFO    plugin/plugin_linux.go:130      plugin has exited with code 0   {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.980+0800    INFO    plugin/plugin_linux.go:133      gorountine of waiting plugin's process will exit        {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.980+0800    INFO    plugin/plugin_linux.go:158      gorountine of sending task to plugin will exit  {"plugin": "driver", "pver": "1.7.0.1", "psign": "8200142de6cefcadacf4bf2d3d459e65a44bf3ad11015a084eeddb053156c7f0"}
2022-06-07T17:21:41.985+0800    INFO    plugin/plugin_linux.go:100      plugin's process will start     {"plugin": "scanner", "pver": "3.0.0.5", "psign": "a59f4a2f9f2ea582e787389536b18f33c340057797764e215f94ea2861d54ada"}
2022-06-07T17:21:42.078+0800    INFO    plugin/plugin.go:209    plugin has been loaded  {"plugin": "scanner", "pver": "3.0.0.5", "psign": "a59f4a2f9f2ea582e787389536b18f33c340057797764e215f94ea2861d54ada"}
2022-06-07T17:21:42.078+0800    INFO    plugin/plugin.go:225    sync done

driver的日志：

[root@aliyun-10-43-28-27 driver]# cat driver.log 
2022-06-07T17:21:41.979871168+08:00     ERROR   driver  src/main.rs:35  when loading kernel module,an error occurred: load module failed: ENOSYS: Function not implemented
[root@aliyun-10-43-28-27 driver]#
0xlwoe21k commented 2 years ago

刚测试了一下，内核版本为2.6.*的重启后driver插件会掉线，就算是重启agent也无法调起。

0xlwoe21k commented 2 years ago

内核中的如下信息，【compat_sys_execve register_kprobe failed】 【 uninstall_kprobe success】这个应该是插入失败了吧

[ELKEID] compat_sys_execve register_kprobe failed, returned -22
[ELKEID] do_init_module register_kprobe failed, returned -22
[ELKEID] SANDBOX: 0
[ELKEID] register_kprobe success: connect_hook: 1,load_module_hook: 1,execve_hook: 1,call_usermodehekoer_hook: 1,bind_hook: 1,create_file_hook: 1,file_permission_hook: 0, ptrace_hook: 1, update_cred_hook: 1, dns_hook: 1, accept_hook:0, mprotect_hook: 0, mount_hook: 1, link_hook: 1, memfd_create: 1, rename_hook: 1,setsid_hook:1, prctl_hook:1, open_hook:0, udev_notifier:1, nanosleep_hook:0, kill_hook: 0, rm_hook: 0,  exit_hook: 0, write_hook: 0, EXIT_PROTECT: 0
[ELKEID] ANTI_ROOTKIT_CHECK: 1
[ELKEID] uninstall_kprobe success
hids_driver: destroy 41 print event class
Percivalll commented 2 years ago

内核中的如下信息，【compat_sys_execve register_kprobe failed】 【 uninstall_kprobe success】这个应该是插入失败了吧

[ELKEID] compat_sys_execve register_kprobe failed, returned -22
[ELKEID] do_init_module register_kprobe failed, returned -22
[ELKEID] SANDBOX: 0
[ELKEID] register_kprobe success: connect_hook: 1,load_module_hook: 1,execve_hook: 1,call_usermodehekoer_hook: 1,bind_hook: 1,create_file_hook: 1,file_permission_hook: 0, ptrace_hook: 1, update_cred_hook: 1, dns_hook: 1, accept_hook:0, mprotect_hook: 0, mount_hook: 1, link_hook: 1, memfd_create: 1, rename_hook: 1,setsid_hook:1, prctl_hook:1, open_hook:0, udev_notifier:1, nanosleep_hook:0, kill_hook: 0, rm_hook: 0,  exit_hook: 0, write_hook: 0, EXIT_PROTECT: 0
[ELKEID] ANTI_ROOTKIT_CHECK: 1
[ELKEID] uninstall_kprobe success
hids_driver: destroy 41 print event class

这个是被手动卸载的，按照我描述的方法发一下完整日志哈，不然看不到上下文没办法定位问题

0xlwoe21k commented 2 years ago

重启agent后的信息

[root@aliyun-10-43-28-27 driver]# service elkeid-agent restart
[root@aliyun-10-43-28-27 driver]# ps -ef | grep elk
root     30639     1  0 09:41 pts/0    00:00:00 /etc/elkeid/elkeid-agent
root     30688 30639  0 09:41 pts/0    00:00:00 /etc/elkeid/plugin/collector/collector
root     30699 30639  0 09:41 pts/0    00:00:00 /etc/elkeid/plugin/journal_watcher/journal_watcher
root     30706 30639  0 09:41 pts/0    00:00:00 /etc/elkeid/plugin/scanner/scanner
root     30801 29636  0 09:47 pts/0    00:00:00 grep elk
[root@aliyun-10-43-28-27 driver]# cat driver.log 
2022-06-07T17:21:41.979871168+08:00     ERROR   driver  src/main.rs:35  when loading kernel module,an error occurred: load module failed: ENOSYS: Function not implemented
2022-06-08T08:48:56.484952537+08:00     ERROR   driver  src/main.rs:35  when loading kernel module,an error occurred: load module failed: ENOSYS: Function not implemented
2022-06-08T09:41:33.078268059+08:00     ERROR   driver  src/main.rs:35  when loading kernel module,an error occurred: load module failed: ENOSYS: Function not implemented
Percivalll commented 2 years ago

rm -rf /etc/elkeid/plugin/driver and restart agent don't manually put the ko file to the dir, but auto download.

0xlwoe21k commented 2 years ago

这样可以解决问题，想请问一下，原因是什么呢？

Percivalll commented 2 years ago

some low kernel verison hasn't finit_module function. By auto downloading, will use insmod command not finit_module function