search

weaveworks / tcptracer-bpf

eBPF program using kprobes to trace TCP events without run-time compilation dependencies
Apache License 2.0
410 stars 61 forks source link

Running two instances of tcptracer-bpf in parallel fails (e.g. Weave Scope + Inspektor Gadget) #70

Open alban opened 4 years ago

alban commented 4 years ago

Running two instances of tcptracer-bpf fails:

In terminal 1:

$ sudo ./tests/tracer

In terminal 2:

$ sudo ./tests/tracer
cannot write "p:ptcp_set_state tcp_set_state\n" to kprobe_events: write /sys/kernel/debug/tracing/kprobe_events: file exists

tcptracer-bpf is used both in Weave Scope and in Inspektor Gadget's Network Policy Advisor and we have an issue (https://github.com/kinvolk/inspektor-gadget/issues/63#issuecomment-616285877) reporting that it fails when both are executed at the same time.

To have several kprobes on the same kernel function, we need either to use a different kprobe names written on /sys/kernel/debug/tracing/kprobe_events, or use anonymous perf-event-fd-based kprobe (https://github.com/iovisor/gobpf/issues/223). The latter is preferable but only works on Linux >= 4.17, so ideally gobpf would implement both fixes.

bboreham commented 4 years ago

Found this when trying to understand my own issue https://github.com/weaveworks/scope/issues/3786

Can we have Scope pick a different set of names to alleviate your problem? Or does it have to be fixed in gobpf ?

alban commented 4 years ago

I looked a bit more into this.

Gobpf currently hard codes this: https://github.com/iovisor/gobpf/blob/master/elf/module.go#L344

eventName := probeType + funcName

And this is not exposed in the API for Scope/IG unfortunately.

But it should be easy to patch gobpf to append a random suffix in the event name.

Bcc adds the suffix _bcc_$PID, which makes it unique enough (tcptracer-bpf runs in the host pid namespace): https://github.com/iovisor/bcc/blob/6e9b4509fc7a063302b574520bac6d49b01ca97e/src/cc/libbpf.c#L961

"%s_bcc_%d", ev_name, getpid());

Gobpf could do the same.