search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.15k stars 4.91k forks source link

[libbeat] Disk queue truncates buffer segments when compression is enabled #39861

Open faec opened 3 months ago

faec commented 3 months ago

Most disk queue internal state, including position and capacity within each queue segment on disk, is calculated in terms of "raw" (uncompressed) bytes. However, on startup, segment lengths (queueSegment.byteCount) are initialized with the on-disk size of the segment files, which will be smaller than the raw size when compression is enabled. In this case, the queue will only read the beginning of each segment file, and the rest will be discarded.

To fix this we probably need to include the logical (uncompressed) size of each segment in the header, like we do with the frame count. Then we can initialize the segment metadata properly when we scan the headers on startup.

elasticmachine commented 3 months ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)