Unable to load eBPF watcher for process event

[teliov]

Context

I am running beyla as a stand alone process using the agent mode as documented here.

When I start beyla like so:

./beyla -config=sample.config

I get the following message:

time=2023-12-13T02:40:09.469Z level=ERROR msg="Unable to load eBPF watcher for process events" component=discover.Watcher interval=5s error="loading and assigning BPF objects: field KprobeSecuritySocketBind: program kprobe_security_socket_bind: map watch_events: map create: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock)"

Config file

open_port: 3111
service_name: api
prometheus_export:
  port: 5009
log_level: INFO
grafana:
  otlp:
    submit: ["metrics"]
otel_metrics_export:
  endpoint: http://localhost:4318
routes:
  patterns:
      - /users/{userId}/ratings
      - /users/{userId}/received-orders
      - /users/{userId}/recipients

System Specs

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

uname -r

5.10.0-23-cloud-arm64

# output truncated
cat /proc/meminfo
MemTotal:        1913524 kB
MemFree:          351592 kB
MemAvailable:     568008 kB
Buffers:           26144 kB

It would seem the rlimit mentioned in the error message is not something I can configure?

[grcevski]

Hi @teliov,

Beyla needs to be run with sudo, I think it's probably the first error we hit with lower permissions.

[teliov]

Hi @grcevski , I was already running as root - so I'd assumed using sudo wouldn't be necessary. But for good measure I ran it with sudo nonetheless and still got the same result.

Also this behaviour is the same for both the beyla binary on the release page, and beyla installed using go install.

[teliov]

Out of curiosity I tested this on debian 12:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Which is running this kernel:

uname -r
6.1.0-15-cloud-arm64

And this glibc version:

ldd --version

ldd (Debian GLIBC 2.36-9+deb12u3) 2.36

If I were to hazard a guess, I'll say the problem is with the glibc version. In this issue: https://github.com/grafana/beyla/issues/390, beyla did complain about glib versions and indeed the glib version on the machines I was testing with did not meet the version requirements beyla wanted.

This debian 12 box however does, and it doesn't give the same error.

[grcevski]

Thanks for letting us know, we'll check to see what's the issue with that exact kernel and OS.

[mzs114]

If you notice the message shared(MEMLOCK may be too low, consider...), this could be due to the setting 'max locked memory' accessible by ulimit -l.

@teliov will it possible to try this with increased limits?

[teliov]

@mzs114 indeed you are right, that seems to be the problem.

On debian 12, the memlock is set to 238752 Kb by default as opposed to 64 Kb on debian 11.

Running beyla after setting ulimit -l 238752 does not give the MEMLOCK may be too low error.

Thanks!

[grcevski]

Thank you @mzs114 ! We should add a documentation section on this.

[pikilisaikiran]

I'm testing beyla on k8s as sidecar container.

I got the same error message

Unable to load eBPF watcher for process events" component=discover.Watcher interval=5s error="loading and assigning BPF objects: field KprobeSecuritySocketBind: program kprobe_security_socket_bind: map watch_events: map create: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock)

The ulimit -l is unlimited on the node on which the pod is running. But it is not inherited to the pod.

worker node details

cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

ulimit -l unlimited

max locked memory in container ulimit -l 64

Seems like we need to change the DockerFile for this limits?

[grcevski]

Hi @pikilisaikiran, yes I guess we can try adding this to our Dockerfile to see if it helps. I think it also might be possible to solve with an initContainers section in k8s, something like (untested):

  initContainers:  
  - name: increase-rlimit
    image: beyla:latest
    command: ["sh", "-c", "ulimit -l unlimited"]
    securityContext:
      privileged: true

[mariomac]

@pikilisaikiran closing the issue due to inactivity. Feel free to reopen it if you think the issue hasn't been solved.

[revathyrams]

Hi, I'm running beyla in a local kubernetes cluster using minikube, following this link. I see in the beyla pod logs that it's unable to load eBPF watcher with the following error:

 level=ERROR msg="Unable to load eBPF watcher for process events" component=discover.ProcessWatcher interval=5s error="loading and assigning BPF objects: field KprobeSecuritySocketBind: program kprobe_security_socket_bind: apply CO-RE relocations: load kernel spec: no BTF found for kernel version 6.4.16-linuxkit: not supported"

This is my docker desktop version 4.24.0 I'm on an M2 mac.

[grcevski]

Hi @revathyrams, this is a limitation of Docker Desktop for Mac. It has a small VM image to emulate the Linux subsystem and they build it without BTF symbols, which we need to tell the offsets of the kernel datastructures. The BTF symbols are on by default for all modern Linux distributions, but Docker is using a custom kernel build.

Best way forward is to look into Colima, they have much better eBPF support for Mac Docker.

[revathyrams]

thanks, I'm able to run Beyla in a k8s cluster with Colima.

[lionelmarksgrafana]

Issue here is Beyla is not capturing traces since no log traffic indicates it..