search

iovisor / bcc

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more
Apache License 2.0
20.63k stars 3.89k forks source link

ringbuffer.submit has a high overhead #3689

Open medislam opened 3 years ago

medislam commented 3 years ago

Is there a method in bcc to send events in batches to user space and not event by event like the ringbuffer.submit() does? I would like to reduce the frequency of using the function ringbuffer.submit() which has an overhead that becomes significant when the rate of the traced events becomes important. Thanks a lot,

chenhengqi commented 3 years ago

Do you have numbers about this high overhead ?

medislam commented 3 years ago

I used a single probe to trace vfs_read, the overhead is up to 30% when the throughput generated by fio is close to 1.2 GB / s (with requests of 4kB). But, if I comment out the ringbuffer.submit function, the overhead will be negligible (1%).

chenhengqi commented 3 years ago

You should do some filtering before submitting your data to ringbuf. ringbuf use spinlock internally, this could causes lock contention. Especially when you trace a function which is expected to be called frequently.

medislam commented 3 years ago

Thank you. But I am looking for if there is a way to store/cache a set of events before sending them by the submit function in BCC ?

chenhengqi commented 3 years ago

That depends on your use-case, you can use per-cpu map/array to accumulate data before you submit. See https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#maps

medislam commented 3 years ago

I tried, but I failed to recover the events in the user space. I haven't found how to format data as a pointer to an array in the callback function.

chenhengqi commented 3 years ago

Post some code snippets here ?

anakryiko commented 3 years ago

Is there a method in bcc to send events in batches to user space and not event by event like the ringbuffer.submit() does? I would like to reduce the frequency of using the function ringbuffer.submit() which has an overhead that becomes significant when the rate of the traced events becomes important. Thanks a lot,

Yes, you can control user-space notifications with extra flags to bpf_ringbuf_submit(). See https://github.com/torvalds/linux/blob/master/include/uapi/linux/bpf.h#L4147-L4149

Combined with bpf_ringbuf_query() you should be able to implement a flexible scheme.