qjerome
commented
3 months ago Hi @miyoyo,
Thank you very much for this very interesting issue and for your detailed analysis.
I think I have identified the issue related to the excessive memory taken by self.task.entry. For the record this is a HashMap of tasks I use for correlation. It turns out that I have a pretty bad cleanup of that map a it never deletes terminated tasks from it, and thus growing indefinitely ! I have to find a smart way to delete useless entries, like terminated tasks with no child. I tested a dirty patch and it seems memory is much more stable if I control well the cleanup. I'll make my best to lend a fix for that issue soon.
Concerning the BytesMut thing it is going to be more challenging to solve as it is the struct holding shared memory with the kernel to hold the kunai eBPF events waiting to be processed by user land. The end-user has granular control over the size of this structure through the --max-buffered-events command line switch or max_buffered_events setting in configuration file. You can have an estimate of the memory taken by this structure by multiplying this value by 3320 (size of today's biggest eBPF events in bytes) multiplied by your number of CPUs of the machine. So for 8 CPUs with the default settings max_buffered_events = 1024 we get 1024 x 3320 x 8 = 27197440 B (~27 MB). This will be more or less un-shrinkable given a max_buffered_events value. On the other hand one knows that it cannot grow indefinitely. If one really wants to decrease the memory taken by this structure, one has to reduce max_buffered_events, at the potential cost of losing events (as the space to store transient events will be reduced). Since v0.2.4 I filter out some disabled events directly in eBPF so they never reach this buffer if it is not needed. So if you really need to save memory through this tuning and do not want to lose events you can combine a custom max_buffered_events value and completely disable some events through configuration. It would be a trade-off you'd have to find between events you want and the memory you want kunai to take.
Cheers,
miyoyo
Description
When Kunai is used very intensely, RAM usage slowly goes up, and does not appear to ever go back down. This usage can go up into multiple hundreds of megabytes, and that memory is only released upon death of the program.
Expected behavior
Kunai has a soft memory cap, or at least goes back down to mostly idle (~20MB) levels over time.
Actual behavior
The RAM usage grows, and never goes down by much.
How to reproduce
Run kunai with no particular arguments Run an event-generating command in a loop, like
while true; do sudo echo 'hi' > /etc/not_important; doneObserve kunai's memory go up via ps, top, or htopHow to profile
Most profilers I have tried just do not work at all with musl, so you have to edit
-muslin xtask/src/user.rs to be-gnuOnce you've done that, build the debug version of kunai (I usedRUSTFLAGS=-g cargo xtask build) Use something like heaptrack to profile the memory usage:heaptrack ./kunaiStart the event-generating command Wait for 10 minutes (memory goes up) Kill the event generating command Wait for 10 minutes (memory does not go down by much) Important: send a SIGKILL (kill -9) to the kunai process. This is to ensure that any unreleased memory is marked as leaked by heaptrack. Observe the heaptrack GUI.My observations
It appears most of the memory allocations that are not released are:
I am not a rust developer, so most of my research here is closer to guesswork than expertise.
Here is my own heaptrack trace as described in
how to profile: heaptrack.kunai.5237.zipOpen it in heaptrack with
heaptrack_gui heaptrack.kunai.5237.zst