'libbpf: load bpf program failed: Invalid argument' when trying to start sysmon on Ubuntu 18.04

[phibid]

Sysmon fails to start on a Ubuntu 18.04 server after installation from package repository:

root@blabla:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:    18.04
Codename:   bionic
root@blabla:/# uname -a
Linux blabla 4.15.0-189-generic #200-Ubuntu SMP Wed Jun 22 19:53:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@blabla:/# dpkg -l | grep -i sysmon
ii  sysmonforlinux                         1.0.2                                           amd64        A system monitor based on eBPF, ported from Windows, that outputs events to Syslog

The log (libbpf dump log truncated, if it is needed please tell me):

Jan 23 11:06:31 blabla systemd[1]: Starting Sysmon event logger...
Jan 23 11:06:31 blabla systemd[1]: Reloading.
Jan 23 11:06:31 blabla systemd[1]: Reloading.
Jan 23 11:06:31 blabla sysmon[18840]: Sysmon v1.0.2 - Monitors system events
Jan 23 11:06:31 blabla sysmon[18840]: Sysinternals - www.sysinternals.com
Jan 23 11:06:31 blabla sysmon[18840]: By Mark Russinovich, Thomas Garnier and Kevin Sheldrake
Jan 23 11:06:31 blabla sysmon[18840]: Copyright (C) 2014-2021 Microsoft Corporation
Jan 23 11:06:31 blabla sysmon[18840]: Using libxml2. libxml2 is Copyright (C) 1998-2012 Daniel Veillard. All Rights Reserved.
Jan 23 11:06:31 blabla sysmon[18840]: Loading configuration file with schema version 4.22
Jan 23 11:06:31 blabla sysmon[18840]: Sysmon schema version: 4.81
Jan 23 11:06:31 blabla sysmon[18840]: Configuration file validated.
Jan 23 11:06:31 blabla sysmon[18840]: Loading configuration file with schema version 4.22
Jan 23 11:06:31 blabla sysmon[18840]: Sysmon schema version: 4.81
Jan 23 11:06:31 blabla sysmon[18840]: Configuration file validated.
Jan 23 11:06:31 blabla sysmon[18840]: Found Kernel version: 4.15
Jan 23 11:06:31 blabla sysmon[18840]: Using EBPF object: .//sysmonEBPFkern4.15.o
Jan 23 11:06:31 blabla sysmon[18840]: libbpf: load bpf program failed: Invalid argument
Jan 23 11:06:31 blabla sysmon[18840]: libbpf: -- BEGIN DUMP LOG ---
Jan 23 11:06:31 blabla sysmon[18840]: libbpf:
....
Jan 23 11:06:31 blabla sysmon[18840]: 1631: (79) r2 = *(u64 *)(r10 -88)
Jan 23 11:06:31 blabla sysmon[18840]: 1632: (0f) r2 += r1
Jan 23 11:06:31 blabla sysmon[18840]: math between map_value pointer and register with unbounded min value is not allowed
Jan 23 11:06:31 blabla sysmon[18840]: libbpf: -- END LOG --
Jan 23 11:06:31 blabla sysmon[18840]: libbpf: failed to load program 'sysmon/ProcCreate/exit'
Jan 23 11:06:31 blabla sysmon[18840]: libbpf: failed to load object './/sysmonEBPFkern4.15.o'
Jan 23 11:06:31 blabla sysmon[18840]: ERROR: failed to load prog: 'Invalid argument'
Jan 23 11:06:31 blabla sysmon[18840]: Telemetry failed to start: eBPF object could not be loaded

Do you know what could be the reason of this failure ? Am I missing something ?

Thanks !

[MarioHewardt]

Hi - thanks for letting me know. I pushed a fix last week that resolved an issue with our eBPF programs failing validation. Could you try latest and see if that resolves it? Note, you will have to build since the fix isn't in any released package yet.

[phibid]

Thanks @MarioHewardt. When can we expect the release of the next .deb package with the fix included?

[MarioHewardt]

The fix I pushed should resolve a slew of these similar eBPF verifier issues. I'd like to get verification that the fix works across distros/kernels before pushing a new package. Would it be possible for you to build and try this out?

[phibid]

Sure, I understand.

So, I have tried to compile on an updated Ubuntu 18.04, however I am stuck during the compilation. I have compiled/installed SysinternalsEBPF with no issue:

# ls -l /usr/lib/libsysinternalsEBPF.so
-rw-r--r-- 1 root root 810552 Jan 25 10:10 /usr/lib/libsysinternalsEBPF.so

Issue occurs during the compilation of SysmonForLinux during make execution. Here the end of the make output:

[ 37%] Checking sysmonEBPFkern4.16.o

eBPF Program Sizes: (max 4096)

sysmon/generic/enter0: 66
sysmon/generic/enter1: 68
sysmon/generic/enter2: 70
sysmon/generic/enter3: 72
sysmon/generic/enter4: 74
sysmon/generic/enter5: 76
sysmon/generic/enter6: 78
sysmon/ProcCreate/exit: 4071
sysmon/FileCreate/exit: 3607
sysmon/FileOpen/exit: 3909
sysmon/FileDelete/exit: 3456
sysmon/FileDeleteAt/exit: 3631
sysmon/FileDeleteAtCwd/exit: 3419
sysmon/sched_process_exit: 241
sysmon/TCPaccept/exit: 208
sysmon/inet_sock_set_state: 191
sysmon/ProcAccessed/exit: 1849
sysmon/consume_skb: 597
sysmon/UDPrecv/exit: 425
sysmon/CloseFD/exit: 49

[ 38%] Checking sysmonEBPFkern4.17-5.1.o

eBPF Program Sizes: (max 4096)

sysmon/generic/rawEnter: 109
sysmon/ProcCreate/rawExit: 4125
  Error: sysmon/ProcCreate/rawExit is greater than max instructions: 4125 > 4096
sysmon/FileCreate/rawExit: 3652
sysmon/FileOpen/rawExit: 3984
sysmon/FileDelete/rawExit: 3489
sysmon/FileDeleteAt/rawExit: 3701
sysmon/FileDeleteAtCwd/rawExit: 3491
sysmon/sched_process_exit: 241
sysmon/TCPaccept/rawExit: 221
sysmon/inet_sock_set_state: 191
sysmon/ProcAccessed/rawExit: 1895
sysmon/consume_skb: 597
sysmon/UDPrecv/rawExit: 443
sysmon/CloseFD/rawExit: 58

CMakeFiles/sysmon.dir/build.make:128: recipe for target 'sysmonEBPFkern4.17-5.1.rep' failed
make[2]: *** [sysmonEBPFkern4.17-5.1.rep] Error 2
CMakeFiles/Makefile2:211: recipe for target 'CMakeFiles/sysmon.dir/all' failed
make[1]: *** [CMakeFiles/sysmon.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Am I missing something ?

Some info regarding the server:

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:    18.04
Codename:   bionic
# uname -a
Linux uyuni03v 4.15.0-202-generic #213-Ubuntu SMP Thu Jan 5 19:19:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[MarioHewardt]

I suspect you may be using an older version of clang/llvm. Try installing 9, that should fix the problem.

[phibid]

Good catch, was able to compile after upgrading clang/llvm to v9. And I can now confirm that your fix seems to have solved the issue as I am now able to start sysmon with no issue.