search

elastic / endpoint

Other
27 stars 10 forks source link

Regression on 'Elastic Endpoint hits "perf" buffer limit on EXECVE monitoring via kprobe when executable command arguments larger than 2048 bytes' #12

Open gpapakyriakopoulos opened 2 years ago

gpapakyriakopoulos commented 2 years ago

As mentioned in the title we observed a regression on one of our previously reported (and fixed) issues, namely #4 on elastic-agent 7.16.2. Errors like the following, along with the kernel taint flag being set to 512, were observed when large enough event traces are generated : 

[Wed Feb  9 19:51:30 2022] ------------[ cut here ]------------
[Wed Feb  9 19:51:30 2022] perf buffer not large enough
[Wed Feb  9 19:51:30 2022] WARNING: CPU: 0 PID: 840 at kernel/trace/trace_event_perf.c:402 perf_trace_buf_alloc+0x86/0x90
[Wed Feb  9 19:51:30 2022] Modules linked in: nfnetlink hid_generic usbhid hid binfmt_misc sr_mod cdrom sg ata_generic ip6table_filter ip6_tables xt_multiport intel_rapl_msr intel_rapl_common xt_pkttype crc32_pclmul xt_tcpudp xt_state xt_conntrack ghash_clmulni_intel iptable_filter aesni_intel ata_piix libaes crypto_simd libata cryptd glue_helper rapl nf_conntrack_tftp nf_conntrack uhci_hcd nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ehci_hcd scsi_mod usbcore virtio_net net_failover joydev virtio_balloon failover pcspkr usb_common i2c_piix4 button qemu_fw_cfg sunrpc loop fuse configfs virtio_rng rng_core ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic virtio_blk bochs_drm drm_vram_helper drm_kms_helper cec drm_ttm_helper ttm drm crct10dif_pclmul crct10dif_common psmouse crc32c_intel evdev serio_raw virtio_pci virtio_ring virtio

These issues are observed on Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux as well as Linux debian 5.10.0-11-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux host instances.

A quick way to reproduce the issue on a host running elastic-agent 7.16.2 and ElasticEndpoint services is to create a large enough file in the filesystem, by running a command such as :

touch /tmp/unix: A*2048 (or enough to exceed the perf buffer)

nicholasberlin commented 2 years ago

Sure would be nice if the kernel's debugfs kprobes just truncated strings instead of throwing warning messages and tainting the kernel, similar to how it handles memory faults. Ok, done complaining, besides there are probably kernel implementation trade offs I'm not aware of or don't understand.

We haven't repro'd this yet, but the example makes sense, so I'll proceed with some thoughts.

4 fixed this issue for execve probes by copying a fixed number of bytes instead of relying on string length. This approach made a certain amount of sense because we copy the program arguments from the new stack's argv blob, and therefore can gather many strings at once.

I think a similar fixed-copy-length approach could work here, assuming there isn't a horrible performance hit. Perhaps the volume of file events and typically short path names will lead to a lot of unnecessary bytes being copied. Maybe it won't matter.

Anyway, we are looking into a fix. Thanks for the bug report.

gpapakyriakopoulos commented 2 years ago

Thanks for the great info! I understand the tradeoffs you described, hopefully you can find a sweet spot between performance/usability and kernel/data safety. Feel free to ask for any additional info that we can provide if needed.

In case it helps, checking the warning in more detail, we saw that this time it is related to an openat() call from kernel/trace/trace_event_perf.c:402 perf_trace_buf_alloc+0x86/0x90. I assume this is a different sys call than the one from the previously reported issue, so I guess this is not a regression after all.