meet ERROR：libbpf: failed to load program 'sysmon/ProcCreate/rawExit'

BlackKD commented 1 year ago

I try to install Sysmon in Centos8，this is my env，but，I meet some ERROR。 if anyone elder met this before？

Linux iZ2ze7nnhgeigrpkeoen64Z 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I install by this way and succes

`CentOS 8

Register Microsoft key and feed sudo rpm -Uvh https://packages.microsoft.com/config/centos/8/packages-microsoft-prod.rpm
Install SysmonForLinux sudo dnf install sysmonforlinux`

But when I run it，I meet this ERROR

`[root@iZ2ze7nnhgeigrpkeoen64Z ~]# sudo sysmon -i

Sysmon v1.0.2 - Monitors system events Sysinternals - www.sysinternals.com By Mark Russinovich, Thomas Garnier and Kevin Sheldrake Copyright (C) 2014-2021 Microsoft Corporation Using libxml2. libxml2 is Copyright (C) 1998-2012 Daniel Veillard. All Rights Reserved.

Job for sysmon.service failed because the control process exited with error code. See "systemctl status sysmon.service" and "journalctl -xe" for details. [root@iZ2ze7nnhgeigrpkeoen64Z ~]# systemctl status sysmon.service ● sysmon.service - Sysmon event logger Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2022-09-23 09:50:50 CST; 3s ago Process: 2123 ExecStart=/opt/sysmon/sysmon -i /opt/sysmon/config.xml -service (code=exited, status=12)

9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: 2149: (b7) r2 = 47 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: 2150: (73) (u8 )(r1 +4095) = r2 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: R0=inv(id=45,smin_value=-4095,smax_value=4095) R1_w=map_value(id=0,off=0,ks=4,vs=8192,smin_value=-4095,smax_value=0) R2_w=inv47 R5=inv0 R6=inv(id> 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: R1 unbounded memory access, make sure to bounds check any such access 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: processed 1320 insns (limit 1000000) max_states_per_insn 2 total_states 109 peak_states 109 mark_read 84 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: libbpf: -- END LOG -- 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: libbpf: failed to load program 'sysmon/ProcCreate/rawExit' 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: libbpf: failed to load object './/sysmonEBPFkern4.17-5.1.o' 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: ERROR: failed to load prog: 'Permission denied' 9月 23 09:50:50 iZ2ze7nnhgeigrpkeoen64Z sysmon[2123]: Telemetry failed to start: eBPF object could not be loaded`

lazysecurity commented 1 year ago

I also appear to have the same problem. I'm running RHEL 8.4 and installed using the Microsoft packages.

# sysmon -accepteula -i /opt/config.xml

Sysmon v1.0.2 - Monitors system events
Sysinternals - www.sysinternals.com
By Mark Russinovich, Thomas Garnier and Kevin Sheldrake
Copyright (C) 2014-2021 Microsoft Corporation
Using libxml2. libxml2 is Copyright (C) 1998-2012 Daniel Veillard. All Rights Reserved.

Loading configuration file with schema version 4.70
Sysmon schema version: 4.81
Configuration file validated.
Job for sysmon.service failed because the control process exited with error code.
See "systemctl status sysmon.service" and "journalctl -xe" for details.

Troubleshooting:

# systemctl status sysmon.service

sysmon.service - Sysmon event logger
   Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2022-10-25 09:05:41 EDT; 14s ago
  Process: 5936 ExecStart=/opt/sysmon/sysmon -i /opt/sysmon/config.xml -service (code=exited, status=12)

<snip>

Oct 25 09:05:41 <hostname redacted> sysmon[5971]: libbpf: failed to load program 'sysmon/ProcCreate/rawExit'
Oct 25 09:05:41 <hostname redacted> sysmon[5971]: libbpf: failed to load object './/sysmonEBPFkern4.17-5.1.o'
Oct 25 09:05:41 <hostname redacted> sysmon[5971]: ERROR: failed to load prog: 'Permission denied'
Oct 25 09:05:41 <hostname redacted> sysmon[5936]: Telemetry failed to start: eBPF object could not be loaded

MarioHewardt commented 1 year ago

I'm working on a fix for this that should take care of the problem (and other similar and duplicate issues filed).

SirStephanikus commented 1 year ago

@MarioHewardt Thanks in advance for that.

However, in the open field the companies are watching sysmon for Linux and they now become pretty angry that RHEL derivates don't work but Ubuntu does ...(and the manual clearly states that RHEL works).

MaliPerica commented 1 year ago

I'm working on a fix for this that should take care of the problem (and other similar and duplicate issues filed).

Is it possible that this issue is related to that some distributions are not having support for BTF which is neccessary for creation of vmlinux.h file (compiled kernel) that is after used by eBPF? Kernel version 4.18 (Debian 10) is having issues with that as I was troubleshooting this. Links below from which I correlated this: https://blog.aquasec.com/vmlinux.h-ebpf-programs https://github.com/aquasecurity/tracee/discussions/713 https://lore.kernel.org/bpf/YdV2NgMG%2FEWwJVQn@kroah.com/T/#r440e35ff5da579680a0c495cd269dc1973395e6e https://github.com/libbpf/libbpf#bpf-co-re-compile-once--run-everywhere https://www.containiq.com/post/btf-bpf-type-format

maketsi commented 1 year ago

Getting the same error on RHEL 8.7, kernel 4.18.0-425.3.1.el8.x86_64

# systemctl status sysmon
● sysmon.service - Sysmon event logger
   Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2022-12-19 16:53:10 EET; 15h ago
  Process: 1448 ExecStart=/opt/sysmon/sysmon -i /opt/sysmon/config.xml -service (code=exited, status=12)

Dec 19 16:53:12 hostname sysmon[1714]: R1 unbounded memory access, make sure to bounds check any such access
Dec 19 16:53:12 hostname sysmon[1714]: processed 1320 insns (limit 1000000) max_states_per_insn 2 total_states 109 peak_states 109 mark_read 84
Dec 19 16:53:12 hostname sysmon[1714]: libbpf: -- END LOG --
Dec 19 16:53:12 hostname sysmon[1714]: libbpf: failed to load program 'sysmon/ProcCreate/rawExit'
Dec 19 16:53:12 hostname sysmon[1714]: libbpf: failed to load object './/sysmonEBPFkern4.17-5.1.o'
Dec 19 16:53:12 hostname sysmon[1714]: ERROR: failed to load prog: 'Permission denied'
Dec 19 16:53:12 hostname sysmon[1448]: Telemetry failed to start: eBPF object could not be loaded
Dec 19 16:53:12 hostname sysmon[1714]: Stopping....
Dec 19 16:53:12 hostname sysmon[1714]: Total events: 0, bad events: 0, ratio = -nan
Dec 19 16:53:12 hostname sysmon[1714]: Lost events: 0, in 0 notifications

Sysmon was installed from microsoft repository (sysinternalsebpf + sysmonforlinux).

MarioHewardt commented 1 year ago

Closing as I've pushed a fix that should resolve the issue. If you encounter it again, please reopen. You may have to run getOffsets (https://github.com/Sysinternals/SysinternalsEBPF/tree/main/getOffsets) to get this to work on CentOS8.

Please note that you will have to build Sysmon until we get new packages out.

Sysinternals / SysmonForLinux