Install sysmon on a Proxmox LXC

[macolinob]

Describe the bug Sysmon installed but service fails to start Ran sysmon -t -i SysmonForLinux-CollectAll-Config.xml and the output is: Job for sysmon.service failed because the control process exited with error code. See "systemctl status sysmon.service" and "journalctl -xeu sysmon.service" for details.

Sysmon version Sysmon v1.1.1

Distro/kernel version Host Proxmox 7.4-3

Guest Ubuntu 22.04.2 Kernel 5.15.104-1-pve

Logs Output of syslog with enough log entries to cover the timespan of the issue. Please run sysmon with the -t switch

Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sched/sched_process_exit': relo https://github.com/Sysinternals/SysmonForLinux/pull/5: insn https://github.com/Sysinternals/SysmonForLinux/pull/117 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'ProcTerminated': found map 7 (perfErrorsMap, sec 13, off 224) for insn https://github.com/Sysinternals/SysmonForLinux/pull/117 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': collecting relocation for section(9) 'tracepoint/sock/inet_soc> Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #0: insn https://github.com/Sysinternals/SysmonForLinux/issues/9 against 'configMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'TCPconnection': found map 0 (configMap, sec 13, off 0) for insn https://github.com/Sysinternals/SysmonForLinux/issues/9 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo https://github.com/Sysinternals/SysmonForLinux/pull/1: insn https://github.com/Sysinternals/SysmonForLinux/issues/22 against 'eventStorageMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'TCPconnection': found map 8 (eventStorageMap, sec 13, off 256) for insn https://github.com/Sysinternals/SysmonForLinux/issues/22 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo https://github.com/Sysinternals/SysmonForLinux/pull/2: insn #144 against 'eventMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'TCPconnection': found map 5 (eventMap, sec 13, off 160) for insn #144 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo https://github.com/Sysinternals/SysmonForLinux/pull/3: insn #157 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'TCPconnection': found map 7 (perfErrorsMap, sec 13, off 224) for insn #157 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo https://github.com/Sysinternals/SysmonForLinux/issues/4: insn #169 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'TCPconnection': found map 7 (perfErrorsMap, sec 13, off 224) for insn #169 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo https://github.com/Sysinternals/SysmonForLinux/pull/5: insn #182 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'TCPconnection': found map 7 (perfErrorsMap, sec 13, off 224) for insn #182 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': collecting relocation for section(11) 'tracepoint/skb/consume_skb' Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #0: insn https://github.com/Sysinternals/SysmonForLinux/issues/10 against 'configMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 0 (configMap, sec 13, off 0) for insn https://github.com/Sysinternals/SysmonForLinux/issues/10 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/pull/1: insn https://github.com/Sysinternals/SysmonForLinux/issues/44 against 'packetStorageMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 3 (packetStorageMap, sec 13, off 96) for insn https://github.com/Sysinternals/SysmonForLinux/issues/44 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/pull/2: insn #420 against 'UDPsendAge' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 4 (UDPsendAge, sec 13, off 128) for insn #420 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/pull/3: insn https://github.com/Sysinternals/SysmonForLinux/issues/441 against 'UDPsendAge' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 4 (UDPsendAge, sec 13, off 128) for insn #441 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #4: insn #449 against 'eventStorageMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 8 (eventStorageMap, sec 13, off 256) for insn #449 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/pull/5: insn #548 against 'eventMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 5 (eventMap, sec 13, off 160) for insn #548 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/issues/6: insn #561 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 7 (perfErrorsMap, sec 13, off 224) for insn #561 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/issues/7: insn #573 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 7 (perfErrorsMap, sec 13, off 224) for insn #573 Apr 05 10:21:50 lino sysmon[47250]: libbpf: sec '.reltracepoint/skb/consume_skb': relo https://github.com/Sysinternals/SysmonForLinux/issues/8: insn #586 against 'perfErrorsMap' Apr 05 10:21:50 lino sysmon[47250]: libbpf: prog 'UDPsend': found map 7 (perfErrorsMap, sec 13, off 224) for insn #586 Apr 05 10:21:50 lino sysmon[47250]: libbpf: Failed to bump RLIMIT_MEMLOCK (err = -1), you might need to do it explicitly! Apr 05 10:21:50 lino sysmon[47250]: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure> Apr 05 10:21:50 lino sysmon[47250]: libbpf: failed to load object './/sysmonEBPFkern5.6-_core.o' Apr 05 10:21:50 lino sysmon[47250]: ERROR: failed to load prog: 'Operation not permitted' Apr 05 10:21:50 lino sysmon[47204]: Telemetry failed to start: eBPF object could not be loaded

Expected behavior To have the service continue to run

[MarioHewardt]

Can you try setting the memlock limits to unlimited?

/etc/security/limits.conf add:

• hard memlock unlimited
• soft memlock unlimited

You will likely need to reboot. Of course, also make sure you run sysmon as root.

[macolinob]

Here is the output with adding the memlock to the limits.conf

" soft memlock unlimited" " hard memlock unlimited"

root@guac:~# sudo sysmon -t -i SysmonForLinux-CollectAll-Config.xml

Sysmon v1.1.1 - Monitors system events Sysinternals - www.sysinternals.com By Mark Russinovich, Thomas Garnier and Kevin Sheldrake Copyright (C) 2014-2023 Microsoft Corporation Licensed under MIT/GPLv2 Using libxml2. libxml2 is Copyright (C) 1998-2012 Daniel Veillard. All Rights Reserved.

Loading configuration file with schema version 4.70 Sysmon schema version: 4.81 Configuration file validated. Job for sysmon.service failed because the control process exited with error code. See "systemctl status sysmon.service" and "journalctl -xe" for details. root@guac:~# journalctl -xe Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sched/sched_process_exit': relo #5: insn #117 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'ProcTerminated': found map 7 (perfErrorsMap, sec 13, off 224) for insn #117 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': collecting relocation for section(9) 'tracepoint/sock/inet_sock> Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #0: insn #9 against 'configMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'TCPconnection': found map 0 (configMap, sec 13, off 0) for insn #9 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #1: insn #22 against 'eventStorageMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'TCPconnection': found map 8 (eventStorageMap, sec 13, off 256) for insn #22 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #2: insn #144 against 'eventMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'TCPconnection': found map 5 (eventMap, sec 13, off 160) for insn #144 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #3: insn #157 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'TCPconnection': found map 7 (perfErrorsMap, sec 13, off 224) for insn #157 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #4: insn #169 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'TCPconnection': found map 7 (perfErrorsMap, sec 13, off 224) for insn #169 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/sock/inet_sock_set_state': relo #5: insn #182 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'TCPconnection': found map 7 (perfErrorsMap, sec 13, off 224) for insn #182 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': collecting relocation for section(11) 'tracepoint/skb/consume_skb' Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #0: insn #10 against 'configMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 0 (configMap, sec 13, off 0) for insn #10 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #1: insn #44 against 'packetStorageMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 3 (packetStorageMap, sec 13, off 96) for insn #44 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #2: insn #420 against 'UDPsendAge' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 4 (UDPsendAge, sec 13, off 128) for insn #420 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #3: insn #441 against 'UDPsendAge' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 4 (UDPsendAge, sec 13, off 128) for insn #441 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #4: insn #449 against 'eventStorageMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 8 (eventStorageMap, sec 13, off 256) for insn #449 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #5: insn #548 against 'eventMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 5 (eventMap, sec 13, off 160) for insn #548 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #6: insn #561 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 7 (perfErrorsMap, sec 13, off 224) for insn #561 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #7: insn #573 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 7 (perfErrorsMap, sec 13, off 224) for insn #573 Apr 06 07:37:04 guac sysmon[1400]: libbpf: sec '.reltracepoint/skb/consume_skb': relo #8: insn #586 against 'perfErrorsMap' Apr 06 07:37:04 guac sysmon[1400]: libbpf: prog 'UDPsend': found map 7 (perfErrorsMap, sec 13, off 224) for insn #586 Apr 06 07:37:04 guac sysmon[1400]: libbpf: Failed to bump RLIMIT_MEMLOCK (err = -1), you might need to do it explicitly! Apr 06 07:37:04 guac sysmon[1400]: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure > Apr 06 07:37:04 guac sysmon[1400]: libbpf: failed to load object './/sysmonEBPFkern5.6-_core.o' Apr 06 07:37:04 guac sysmon[1400]: ERROR: failed to load prog: 'Operation not permitted' Apr 06 07:37:04 guac sysmon[1361]: Telemetry failed to start: eBPF object could not be loaded

[juju4]

If you are in lxc, you may need to allow some capabilities and may be in worse case, make container privileged which is better to avoid. but I did not identified such need from basic testing in docker in my ansible role https://github.com/juju4/ansible-sysmon/ and I more cap restraints in my systemd config https://github.com/juju4/ansible-sysmon/blob/main/templates/systemd-hardening.conf.j2#L50

the only thing needed on my side was CAP_BPF CAP_PERFMON

[MarioHewardt]

libbpf (Sysmon dependency) is failing when attempting to set MEMLOCK resource limits using setrlimit. More specifically, it is failing with an EPERM (operation not permitted). I'm not super familiar with ProxMox LXC but it seems that in that environment for some reason it is not allowed to set the resource (hard) limits. There is a similar problem using Docker where hard limits cannot be set but you can specify the --ulimit switch at the point of running the container to set limits that way. Perhaps there is something similar in the ProxMox environment. As a test, you can try setting the hard limit using something like - ulimit -Hl 500 and see if you get the same error.

[MarioHewardt]

Hi - Closing this for now. Please feel free to re-open with the results of the above test.