arm64 support: eBPF monitoring mode

redanaheim commented 1 month ago

Describe the bug

The eBPF monitoring mode cannot be used successfully on arm64.

OpenSnitch version - 1.6.5.1
OS: NixOS
Version: 24.11.20240524 Vicuña
Window Manager: GNOME shell
Kernel version: Linux asahimbp 6.8.9-asahi #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan 1 00:00:00 UTC 1980 aarch64 GNU/Linux

I had been using OpenSnitch in proc monitoring mode but started using a WireGuard VPN, so I wished to enable eBPF monitoring mode as per the FAQ.

I checked the requirements using opensnitchd -check-requirements and added all the required kernel configuration options my NixOS configuration:

 # List of options required to enable: https://github.com/evilsocket/opensnitch/issues/774
  boot.kernelPatches = [
    {
      name = "opensnitch_enable";
      patch = null;
      extraStructuredConfig = {
        FTRACE = lib.kernel.yes; # yes
        KPROBES = lib.kernel.yes; # yes
        KPROBES_ON_FTRACE = lib.kernel.yes;
        HAVE_KPROBES = lib.kernel.yes; # yes
        HAVE_KPROBES_ON_FTRACE = lib.kernel.yes;
        KPROBE_EVENTS = lib.kernel.yes; # yes
        HAVE_SYSCALL_TRACEPOINTS = lib.kernel.yes; # yes
        FTRACE_SYSCALLS = lib.kernel.yes; # yes
        UPROBE_EVENTS = lib.kernel.yes; # yes
        INET_DIAG = lib.kernel.yes; # yes
        INET_TCP_DIAG = lib.kernel.yes; # yes
        INET_UDP_DIAG = lib.kernel.yes; # yes
        INET_DIAG_DESTROY = lib.kernel.yes; # yes
      };
    }
  ];

I confirmed after rebuilding my configuration that the configuration options were listed in /proc/config.gz (that's what the comment after each option is). Unfortunately, I discovered that HAVE_KPROBES_ON_FTRACE and KPROBES_ON_FTRACE were both not present (much less y). I initially thought that they were overwritten by configuration options from nixos-apple-silicon, but after combing through their files and the Asahi Linux kernel config, neither were present.

As it turns out, HAVE_KPROBES_ON_FTRACE is determined unset because the kernel on arm64 does not yet support KPROBES_ON_FTRACE (source: https://github.com/torvalds/linux/blob/master/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt).

Therefore, I am unable to set the required configuration options on my Apple Silicon machine and opensnitchd -check-requirements displays the following:

        Checking => CONFIG_KPROBES=y
        Checking => CONFIG_KPROBES_ON_FTRACE=y
         - KPROBES not fully supported by this kernel.
        Checking => CONFIG_KPROBES_ON_FTRACE=y
         - KPROBES not fully supported by this kernel.
        Checking => CONFIG_HAVE_KPROBES=y
        Checking => CONFIG_HAVE_KPROBES_ON_FTRACE=y
         - KPROBES not fully supported by this kernel.
        Checking => CONFIG_KPROBE_EVENTS=y

        * kprobes        ✘

While the daemon produces the following error message:

May 26 18:15:12 asahimbp opensnitchd[1429]: [2024-05-26 22:15:12]  IMP  Starting opensnitch-daemon v1.6.5.1
May 26 18:15:12 asahimbp opensnitchd[1429]: [2024-05-26 22:15:12]  INF  Loading configuration file /nix/store/dxy0l2g0c3wah9nw29sw98vh8d77f198-default-config.json ...
May 26 18:15:12 asahimbp opensnitchd[1429]: [2024-05-26 22:15:12]  INF  Loading rules from /var/lib/opensnitch/rules ...
May 26 18:15:13 asahimbp opensnitchd[1429]: found /nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6
May 26 18:15:13 asahimbp opensnitchd[1429]: [2024-05-26 22:15:13]  WAR  EBPF-DNS: Failed to attach uprobe uretprobe/gethostbyname : cannot write "r:r___nix_store_27fg1mkiymj2b344j80kygsbxfcdl5qi_glibc_2_39_52_lib_libc_so_6_114ca0_gobpf_1429 /nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6:0x114ca0\n" to uprobe_events: write /sys/kernel/debug/tracing/uprobe_events: invalid argument, (/nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6, 1133728)
May 26 18:15:13 asahimbp opensnitchd[1429]: [2024-05-26 22:15:13]  WAR  EBPF-DNS: Failed to attach uprobe uprobe/getaddrinfo : cannot write "p:p___nix_store_27fg1mkiymj2b344j80kygsbxfcdl5qi_glibc_2_39_52_lib_libc_so_6_112a90_gobpf_1429 /nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6:0x112a90\n" to uprobe_events: write /sys/kernel/debug/tracing/uprobe_events: invalid argument, (/nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6, 1125008)
May 26 18:15:13 asahimbp opensnitchd[1429]: [2024-05-26 22:15:13]  WAR  EBPF-DNS: Failed to attach uprobe uretprobe/getaddrinfo : cannot write "r:r___nix_store_27fg1mkiymj2b344j80kygsbxfcdl5qi_glibc_2_39_52_lib_libc_so_6_112a90_gobpf_1429 /nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6:0x112a90\n" to uprobe_events: write /sys/kernel/debug/tracing/uprobe_events: invalid argument, (/nix/store/27fg1mkiymj2b344j80kygsbxfcdl5qi-glibc-2.39-52/lib/libc.so.6, 1125008)
May 26 18:15:13 asahimbp opensnitchd[1429]: [2024-05-26 22:15:13]  WAR  EBPF-DNS: Failed to find symbols for uprobes.
May 26 18:15:13 asahimbp opensnitchd[1429]: [2024-05-26 22:15:13]  WAR  EBPF-DNS: Unable to attach ebpf listener: Failed to find symbols for uprobes

Side note - I don't understand why the error messages involve failure to attack uprobes while the uprobe kernel configuration options (and every other option, in fact) appear to be fine:

        Checking => CONFIG_UPROBES=y
        Checking => CONFIG_UPROBE_EVENTS=y

        * uprobes        ✔

Side note number 2 for which I will submit a PR: the check for CONFIG_KPROBES_ON_FTRACE is duplicated at https://github.com/evilsocket/opensnitch/blob/03747ea0e3efe2bd7274bbc8ee7039f97b80f861/daemon/core/system.go#L88.

To Reproduce

Steps to reproduce the behavior:

Attempt to set the CONFIG_KPROBES_ON_FTRACE option to true on an arm64 machine.
Realize it doesn't work because it's not supported yet.
Try restarting opensnitchd.service with eBPF monitoring mode enabled anyway, and when it displays a bewildering error message try running opensnitchd -check-requirements.

Expected behavior (optional) arm64 supported with eBPF mode so that WireGuard connections can be filtered correctly.

gustavo-iniguez-goya commented 1 month ago

Hi @redanaheim ,

Thank you for reporting this issue.

Could you post the log lines indicating if any of the modules has been loaded? grep opensnitch.*o /var/log/opensnitchd.log, it should output something like:

[2024-05-27 19:13:19]  DBG  [eBPF] trying to load /usr/lib/opensnitchd/ebpf/opensnitch-procs.o
[2024-05-27 19:13:25]  INF  [eBPF] module loaded: /usr/lib/opensnitchd/ebpf/opensnitch-procs.o

Is there any entry in /sys/kernel/debug/tracing/kprobe_events?

I tested this on an aarch64 VM, ubuntu 20.04, kernel 5.8.0, and the modules load and work, although the behaviour is a bit erratic, for opesnitch-procs.o mainly.

I'll try to test it on another distro/hardware/kernel.

As it turns out, HAVE_KPROBES_ON_FTRACE is determined unset because the kernel on arm64 does not yet support KPROBES_ON_FTRACE (source: https://github.com/torvalds/linux/blob/master/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt).

thank you for this info!

Side note - I don't understand why the error messages involve failure to attack uprobes while the uprobe kernel configuration options (and every other option, in fact) appear to be fine:

That problem seems to be the same than #1013 , particular to NixOS.

gustavo-iniguez-goya commented 1 month ago

tested on a raspberry pi with kernel 6.1.21-v8+, ebpf network interception works, including the hook for VPNs/tunnels:

# cat /sys/kernel/debug/tracing/kprobe_events 
p:kprobes/pudp_sendmsg udp_sendmsg
p:kprobes/pudpv6_sendmsg udpv6_sendmsg
p:kprobes/piptunnel_xmit iptunnel_xmit
p:kprobes/ptcp_v4_connect tcp_v4_connect
r10:kprobes/rtcp_v4_connect tcp_v4_connect
p:kprobes/ptcp_v6_connect tcp_v6_connect
r10:kprobes/rtcp_v6_connect tcp_v6_connect

[2024-05-29 14:26:22]  DBG  new connection tcp => 44070:192.168.0.13 -> 1.1.1.1 (one.one.one.one):443 uid: 0, mark: 0
[2024-05-29 14:26:22]  DBG  [ebpf conn] not in cache, but in execEvents: tcp44070192.168.0.131.1.1.1443, 4482 -> /usr/bin/curl
[2024-05-29 14:26:22]  DBG  [ebpf conn] adding item to cache: tcp44070192.168.0.131.1.1.1443

syscall execve hook doesn't work, but sched_process_exit do (so I guess sched_process_exec would work as well)

[2024-05-29 14:26:22]  DBG  [eBPF exit event] -> 4482
[2024-05-29 14:26:22]  DBG  [eBPF exit event inCache] -> 4482
[2024-05-29 14:26:22]  DBG  [eBPF exit event] -> 4482

kernel config / requirements

``` Checking system requirements for kernel version 6.1.21-v8+ ------------------------------------------------------------------------------ Checking => CONFIG_KPROBES=y Checking => CONFIG_KPROBES_ON_FTRACE=y - KPROBES not fully supported by this kernel. Checking => CONFIG_KPROBES_ON_FTRACE=y - KPROBES not fully supported by this kernel. Checking => CONFIG_HAVE_KPROBES=y Checking => CONFIG_HAVE_KPROBES_ON_FTRACE=y - KPROBES not fully supported by this kernel. Checking => CONFIG_KPROBE_EVENTS=y * kprobes ✘ Checking => CONFIG_UPROBES=y * UPROBES not supported. Common error => cannot open uprobe_events: open /sys/kernel/debug/tracing/uprobe_events Checking => CONFIG_UPROBE_EVENTS=y * UPROBES not supported. Common error => cannot open uprobe_events: open /sys/kernel/debug/tracing/uprobe_events * uprobes ✘ Checking => CONFIG_FTRACE=y * ftrace ✔ Checking => CONFIG_HAVE_SYSCALL_TRACEPOINTS=y Checking => CONFIG_FTRACE_SYSCALLS=y - CONFIG_FTRACE_SYSCALLS or CONFIG_HAVE_SYSCALL_TRACEPOINTS not set. Common error => error enabling tracepoint tracepoint/syscalls/sys_enter_execve: cannot read tracepoint id * syscalls ✘ Checking => CONFIG_NETFILTER_NETLINK_QUEUE=[my] Checking => CONFIG_NFT_QUEUE=[my] Checking => CONFIG_NETFILTER_XT_TARGET_NFQUEUE=[my] * nfqueue ✔ Checking => CONFIG_NETFILTER_NETLINK=[my] Checking => CONFIG_NETFILTER_NETLINK_QUEUE=[my] Checking => CONFIG_NETFILTER_NETLINK_ACCT=[my] * netlink ✔ Checking => CONFIG_INET_DIAG=[my] Checking => CONFIG_INET_TCP_DIAG=[my] Checking => CONFIG_INET_UDP_DIAG=[my] * One or more socket monitoring interfaces are not enabled (CONFIG_INET_DIAG, CONFIG_INET_TCP_DIAG, CONFIG_INET_UDP_DIAG, CONFIG_DIAG_DESTROY (Reject feature)). Checking => CONFIG_INET_DIAG_DESTROY=[my] * One or more socket monitoring interfaces are not enabled (CONFIG_INET_DIAG, CONFIG_INET_TCP_DIAG, CONFIG_INET_UDP_DIAG, CONFIG_DIAG_DESTROY (Reject feature)). * net diagnostics ✘ ```

evilsocket / opensnitch

arm64 support: eBPF monitoring mode #1138