Queue keeps stale event data in memory in 8.15

[faec]

In 8.15, events in the memory queue are not freed when they are acknowledged (as intended), but only when they are overwritten by later events in the queue buffer. This means for example if a configuration has a queue size of 5000, but the input data is low-volume and only 100 events are active at once, then the queue will gradually store more events until reaching 5000 in memory at once, then start replacing those with new events. This means:

• the worst memory increase is for low-throughput configs with large queues.
• for users whose queues were already sized proportionate to their throughput, memory use is increased but only marginally (probably still better than e.g. 8.12-13). This is also probably why it was missed, since most of our benchmarks focus on high-throughput scenarios.
• users that are affected should be able to mitigate it by lowering their queue size.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)