iovisor / bcc

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more
Apache License 2.0
20.58k stars 3.88k forks source link

Stuck when attaching kretprobe on _raw_write_lock #2300

Open prathyushpv opened 5 years ago

prathyushpv commented 5 years ago

I was experimenting with bcc on my ubuntu 18.04 machine. The machine is getting stuck when I execute the following code

from bcc import BPF BPF(text='int kretprobe___raw_spin_lock(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

Once above script starts executing the machine is stuck. I don't know if I am doing anything wrong.

But there is no problem when I run the following code. I just changed kretprobe to kprobe.

from bcc import BPF BPF(text='int kprobe___raw_spin_lock(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()

yonghong-song commented 5 years ago

it is possible that raw_spin_lock is called inside the bpf program helper bpf_trace_printk and due to this some kind of deadlock may happen?

prathyushpv commented 5 years ago

But it is possible to insert kprobe in _raw_spin_lock. The problem is only for kretprobe.

yonghong-song commented 5 years ago

At the beginning of _raw_spin_lock, the lock has not been held. At the return of _raw_spin_lock(), the lock is held. This could cause deadlock if bpf_trace_printk caused _raw_spin_lock is called again.

prathyushpv commented 5 years ago

If that is the case, using ebpf this way is crashing the system. ebpf has to be safe. Isn't it a problem if an ebpf script can crash the system?

yonghong-song commented 5 years ago

I guess the reason is bpf_trace_printk. If you use this one, you will get a kernel warning. bpf_trace_printk is not recommended for production use.

prathyushpv commented 5 years ago

I tried using perf_submit() instead of bpf_trace_printk. Still the same problem!

yonghong-song commented 5 years ago

Maybe perf_submit() (eventually calling bpf_perf_event_output helper) also uses raw_spin_lock somewhere (linux:kernel/events/core.c). Could you try a do nothing bpf program? Looks like tracing raw_spin_lock with kretprobe just not safe.

prathyushpv commented 5 years ago

A do nothing ebpf program is also crashing the system

from bcc import BPF
BPF(text='int kretprobe___raw_spin_lock(void *ctx) { return 0; }')