ARM64: sockaddr_un size checks verifier problem

[rafaeldtinoco]

Both probes: kprobe/security_socket_bind and kprobe/security_socket_accept, have the following snippet:

else if (sa_fam == AF_UNIX) {
    if (addr_len <= sizeof(struct sockaddr_un)) {
        struct sockaddr_un sockaddr = {};
        sockaddr.sun_family = READ_KERN(address->sa_family);
        bpf_probe_read(&sockaddr, addr_len, (void *)address);
        save_to_submit_buf(&data, (void *)&sockaddr, sizeof(struct sockaddr_un), 1);
    }
    else save_to_submit_buf(&data, (void *)address, sizeof(struct sockaddr_un), 1);
}

and the bpf_probe_read() is not passing verifier in arm64 5.13 kernel:

809: (7b) *(u64 *)(r10 -232) = r1
810: (7b) *(u64 *)(r10 -240) = r1
811: (7b) *(u64 *)(r10 -248) = r1
812: (7b) *(u64 *)(r10 -256) = r1
813: (7b) *(u64 *)(r10 -264) = r1
814: (7b) *(u64 *)(r10 -272) = r1
815: (bf) r1 = r10
816: (07) r1 += -272
; bpf_probe_read(&sockaddr, addr_len, (void *)address);
817: (bf) r2 = r6
818: (79) r3 = *(u64 *)(r10 -280)
819: (85) call bpf_probe_read#4
R2 min value is negative, either use unsigned or 'var &= const'
processed 705 insns (limit 1000000) max_states_per_insn 0 total_states 43 peak_states 43 mark_read 33

libbpf: -- END LOG --
libbpf: failed to load program 'trace_security_socket_connect'
libbpf: failed to load object 'embedded-core'
2021/11/08 21:22:35 error creating Tracee: failed to load BPF object

$ uname -a Linux ubuntum1 5.13.0-20-generic #20-Ubuntu SMP Fri Oct 15 14:24:24 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

[rafaeldtinoco]

This type of verifier error has been observed in some other projects as well:

https://github.com/iovisor/bcc/issues/1260

and it is related to passing non fixed length to bpf helpers (support has been added at https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/kernel/bpf/verifier.c?id=06c1c049721a995dee2829ad13b24aaf5d7c5cce and fixed in a later commit).

It is funny that we're only facing in aarch64 environments. Perhaps the code generated by aarch64 LLVM is not satisfying eBPF verifier in kernel 5.13 (register re-use made verifier not to be able to speculate boundaries).

eBPF verifier is complaining about potential boundary bad access due to:

bpf_probe_read(&sockaddr, addr_len, (void *)address);

[rafaeldtinoco]

When trying to mitigate the issue:

diff --git a/tracee-ebpf/tracee/tracee.bpf.c b/tracee-ebpf/tracee/tracee.bpf.c
index b1e3ed1..67e6559 100644
--- a/tracee-ebpf/tracee/tracee.bpf.c
+++ b/tracee-ebpf/tracee/tracee.bpf.c
@@ -3047,12 +3047,15 @@ int BPF_KPROBE(trace_security_socket_connect)
         save_to_submit_buf(&data, (void *)address, sizeof(struct sockaddr_in6), 1);
     }
     else if (sa_fam == AF_UNIX) {
+#if defined(__TARGET_ARCH_x86) // TODO: this is broken in arm64 (issue: #1129)
         if (addr_len <= sizeof(struct sockaddr_un)) {
             struct sockaddr_un sockaddr = {};
             bpf_probe_read(&sockaddr, addr_len, (void *)address);
             save_to_submit_buf(&data, (void *)&sockaddr, sizeof(struct sockaddr_un), 1);
         }
-        else save_to_submit_buf(&data, (void *)address, sizeof(struct sockaddr_un), 1);
+        else
+#endif
+            save_to_submit_buf(&data, (void *)address, sizeof(struct sockaddr_un), 1);
     }

     return events_perf_submit(&data, SECURITY_SOCKET_CONNECT, 0);
@@ -3164,12 +3167,15 @@ int BPF_KPROBE(trace_security_socket_bind)
         }
     }
     else if (sa_fam == AF_UNIX) {
+#if defined(__TARGET_ARCH_x86) // TODO: this is broken in arm64 (issue: #1129)
         if (addr_len <= sizeof(struct sockaddr_un)) {
             struct sockaddr_un sockaddr = {};
             bpf_probe_read(&sockaddr, addr_len, (void *)address);
             save_to_submit_buf(&data, (void *)&sockaddr, sizeof(struct sockaddr_un), 1);
         }
-        else save_to_submit_buf(&data, (void *)address, sizeof(struct sockaddr_un), 1);
+        else
+#endif
+            save_to_submit_buf(&data, (void *)address, sizeof(struct sockaddr_un), 1);
     }

     if (connect_id.port) {

there is also another issue appearing:

$ sudo ./dist/tracee-ebpf --debug --trace 'event!=sched*'
OSInfo: PRETTY_NAME: "Ubuntu 21.10"
OSInfo: VERSION_ID: "21.10"
OSInfo: VERSION: "21.10 (Impish Indri)"
OSInfo: VERSION_CODENAME: impish
OSInfo: ID: ubuntu
OSInfo: ID_LIKE: debian
OSInfo: KERNEL_RELEASE: 5.13.0-21-generic
OSInfo: ARCH: arm64
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
2021/11/30 13:52:40 error creating Tracee: failed to update map sys_exit_tails

and it does not appear when selecting other specific events such as openat, openat2:

$ sudo ./dist/tracee-ebpf --debug --trace 'event=openat,openat2'
OSInfo: PRETTY_NAME: "Ubuntu 21.10"
OSInfo: VERSION_ID: "21.10"
OSInfo: VERSION: "21.10 (Impish Indri)"
OSInfo: VERSION_CODENAME: impish
OSInfo: ID: ubuntu
OSInfo: ID_LIKE: debian
OSInfo: KERNEL_RELEASE: 5.13.0-21-generic
OSInfo: ARCH: arm64
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
13:53:12:010991  0      systemd          1       1       40               openat               dirfd: -100, pathname: /proc/319/cgroup, flags: O_RDONLY|O_CLOEXEC, mode: 0
2021-11-30 13:53:12.853531672 -0300 -03  swapper/0         0        debug_net/inet_sock_set_state  LocalIP: 0.0.0.0, LocalPort: 22, RemoteIP: 0.0.0.0, RemotePort: 0, Protocol: 6, OldState: 10, NewState: 3, SockPtr: 0xffff0000
2021-11-30 13:53:12.85360688 -0300 -03  swapper/0         0        debug_net/inet_sock_set_state  LocalIP: 10.211.55.4, LocalPort: 22, RemoteIP: 10.211.55.2, RemotePort: 65155, Protocol: 6, OldState: 3, NewState: 1, SockPtr: 0xffff0000
13:53:12:854856  0      sshd             47969   47969   10               openat               dirfd: -100, pathname: /proc/self/oom_score_adj, flags: O_WRONLY|O_CREAT|O_TRUNC, mode: 438

[yanivagman]

@rafaeldtinoco Do you use CO-RE in this environment you use for ARM64? If so, this might be related: https://github.com/aquasecurity/tracee/issues/1189#issuecomment-983727608

[itaysk]

is this still relevant?

[rafaeldtinoco]

This is relevant but we've got a workaround in place. I didn't want to close this, but won't have my name assigned to it just now (as they are bigger priorities).

[itaysk]

@rafaeldtinoco what's the status of this?

[rafaeldtinoco]

@rafaeldtinoco what's the status of this?

Still in place. Never revisited. Fix was done by commit 48654aa19, mitigated by commit a298a640f and stayed like that. I believe arm64 verifier will probably continue complaining about that logic and we should do what is being done in a way all verifiers accept. WE did not have arm64 tests back then in the workflows (so we could not enforce a fix at that time).