Sysmon randomly fails to start on Ubuntu 18.04

[phibid]

I have noticed that Sysmon randomly failed to start on a Ubuntu 18.04:

# systemctl stop sysmon
# systemctl start sysmon
Job for sysmon.service failed because the control process exited with error code.
See "systemctl status sysmon.service" and "journalctl -xe" for details.
# systemctl status sysmon
● sysmon.service - Sysmon event logger
   Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2023-02-08 18:08:59 UTC; 3s ago
  Process: 17881 ExecStart=/opt/sysmon/sysmon -i /opt/sysmon/config.xml -service (code=exited, status=13)
 Main PID: 94573 (code=exited, status=0/SUCCESS)

Feb 08 18:08:57  sysmon[17881]: Using EBPF object: .//sysmonEBPFkern4.15.o
Feb 08 18:08:58  sysmon[17881]: Discovering offsets...mm_arg_start offset not found
Feb 08 18:08:58  sysmon[17881]: Get Offsets Error: Discovery - could not find the MM offsets
Feb 08 18:08:59  sysmon[17881]: Could not automatically discover kernel offsets.
Feb 08 18:08:59  sysmon[17881]: Build and run the get_offsets module to generate the offsets config file:
Feb 08 18:08:59  sysmon[17881]: /opt/sysinternalsEBPF/sysinternalsEBPF_offsets.conf
Feb 08 18:08:59  sysmon[17881]: Telemetry failed to start: Configuration could not be loaded
Feb 08 18:08:59  systemd[1]: sysmon.service: Control process exited, code=exited status=13
Feb 08 18:08:59  systemd[1]: sysmon.service: Failed with result 'exit-code'.
Feb 08 18:08:59  systemd[1]: Failed to start Sysmon event logger.
# 
# systemctl start sysmon

The second start correctly worked.

[MarioHewardt]

Hi - could you try running the get_offsets module as per https://github.com/Sysinternals/SysinternalsEBPF/tree/main/getOffsets and see if you can reproduce?

[phibid]

Seems that it works. However, as was not able to compile this in our prod server, so had to compile it on a different server (same OS) and copy the file there. Is this a correct way to do ?

[MarioHewardt]

Yep, if you are not able to run get_offsets on the prod machine, running it on a kernel compatible system works. I'm tagging this as a bug to investigate why the auto discovery process fails on that specific kernel. Can you update with the exact kernel version?

[phibid]

Ok, thanks. Here it is 4.15.0-189-generic.

[MarioHewardt]

@phibid - Can you try with the latest version (1.1) and see if this reproduces? I'm unable to reproduce it now on a clean 18.04 with Sysmon 1.1

[phibid]

@MarioHewardt Seems more stable than before, however still can observe some failed restart some time to time:

# dpkg -l | grep sysmon
ii  sysmonforlinux                         1.1.0                                           amd64        A system monitor based on eBPF, ported from Windows, that outputs events to Syslog

# systemctl status sysmon
● sysmon.service - Sysmon event logger
   Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2023-03-14 07:53:35 UTC; 3s ago
  Process: 23233 ExecStart=/opt/sysmon/sysmon -i /opt/sysmon/config.xml -service (code=exited, status=13)
 Main PID: 22582 (code=exited, status=0/SUCCESS)

Mar 14 07:53:35 xx sysmon[23233]: Discovery process: auto discovery
Mar 14 07:53:35 xx sysmon[23233]: Discovering offsets...mm_arg_start offset not found
Mar 14 07:53:35 xx sysmon[23233]: Get Offsets Error: Discovery - could not find the MM offsets
Mar 14 07:53:35 xx sysmon[23233]: Could not automatically discover kernel offsets.
Mar 14 07:53:35 xx sysmon[23233]: Build and run the get_offsets module to generate the offsets config file:
Mar 14 07:53:35 xx sysmon[23233]: /opt/sysinternalsEBPF/sysinternalsEBPF_offsets.conf
Mar 14 07:53:35 xx sysmon[23233]: Telemetry failed to start: Configuration could not be loaded
Mar 14 07:53:35 xx systemd[1]: sysmon.service: Control process exited, code=exited status=13
Mar 14 07:53:35 xx systemd[1]: sysmon.service: Failed with result 'exit-code'.
Mar 14 07:53:35 xx systemd[1]: Failed to start Sysmon event logger.

[MarioHewardt]

Hi - Closing as 18.04 has gone EOL.