Open szibis opened 4 months ago
Even after kpod redeployment old buffers are not flushed from disks
I have no name!@vector-metrics-egress-us-west-2c-0:/data_dir/buffer/v2/datadog_agent_misc_metrics$ ls -lah
total 552M
drwxrwsr-x 2 1000 1000 4.0K Apr 17 08:57 .
drwxrwsr-x 4 1000 1000 4.0K Apr 15 11:17 ..
-rw-rw---- 1 1000 1000 125M Apr 16 14:48 buffer-data-15.dat
-rw-rw---- 1 1000 1000 128M Apr 16 16:33 buffer-data-18.dat
-rw-rw---- 1 1000 1000 126M Apr 15 19:37 buffer-data-1.dat
-rw-rw---- 1 1000 1000 126M Apr 17 04:06 buffer-data-29.dat
-rw-rw---- 1 1000 1000 48M Apr 17 08:54 buffer-data-41.dat
-rw-rw---- 1 1000 1000 24 Apr 17 08:59 buffer.db
-rw-r--r-- 1 1000 1000 0 Apr 17 08:57 buffer.lock
Now two days of buffers still on disks
ok after redeploy number of events drop properly, but buffer remain on disks - this means that after send they are not cleaned ??
Thanks for this report @szibis . I'm not actually sure when the buffer files are deleted when events are processed. @tobz is this something you know off the top of your head?
Maybe they are not processed when reaching disk buffer on overflow and thats why the files are not removed ??
A note for the community
Problem
Two problems:
When we look into one of the kpods we see that disk buffer is used and growing just like showed on graphs.
No additional info in logs.
Configuration
Version
0.37
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response