[Proposal] Memory Leak detection system

[timroes]

We have seen a histry of memory leaks in Kibana (#99473, #99471, elastic/elastic-charts#1148, #20342, #59454), which are often impacting users running Kibana fo ra long time, especially with auto refresh on. This might cause the browser tab to crash then after a couple of days running. While we can't reliably automate memory leak detection during development, I'd suggest we build a system that tries to at least detect if a recent snapshot introduced a memory leak.

That system could do the following:

• Run our most recent snapshot of Kibana/ES in a VM.
• Inject a couple of test dashboards (containing a broad range of visualizations, maps, and other embeddables) into Kibana.
• Have them all running in headless Chrome instances with auto-refresh set to a rather low value (5-10s).
• Monitor the a couple of key metrics (memory, DOM node count, registered listeners, ..) via the instrumentation APIs (which hopefully will give access to that) over time and log them (into an ES obviously ;) ).
• Set up some alerting that will detect if any of the metrics is not properly being "garbage collected", but simply counts up over time. This would be a high indicator of a memory leak in Kibana.
• Redeploy that system with a new snapshot e.g. every 2 days or another time frame we feel confident that the memory leak must have clearly shown in charts over that time.

This would of course not allow us to pinpoint a memory leak directly to a PR, which would be nice. But it would help us at least recognize after 2 days if some change caused us a new memory leak in Kibana, and someone can start investigating the snapshot manually finding the (newly introduced) problematic code.

cc @pauldotpower

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[LeeDr]

@marius-dr did this a few times on an ad-hoc basis. I think we just had Stack Monitoring on the Kibana server and metricbeat data collection on the browser process CPU and memory. Maybe a bit more. @marius-dr could you describe what you did here? Was the monitoring data and metricbeat data sent to a different cluster? I think we need that so we can compare over time.

[Dosant]

Was debugging my branch thinking that my changes caused a memory leak, turned out it exists in main. Filed coupe more issues:

• https://github.com/elastic/kibana/issues/120251 (this one a recent regression. Before this Discover's tree was properly cleaned up)
• https://github.com/elastic/kibana/issues/120253
• https://github.com/elastic/kibana/issues/120254