Introduce performance regression tests for pipeline initialization/compilation

[jsvd]

Pipeline compilation has many moving parts, from generating PipelineIR, to code generation, to code compilation with Janino. Logstash already has tests that show that the execution flow works as expected but we're lacking the oversight on the time pipelines take to start, especially in concurrent situations like multiple pipelines + multiple workers.

[roaksoax]

In https://github.com/elastic/logstash/issues/11175 , we discussed adding a test such as "Add integration harness around the expected compilation time which should be similar between single or multiple workers."

The idea of the test was to hardcode a expected compilation time to identify potential performance regressions. From this we know what the compilation time should be, and by adding this test, we would know for sure that if the compilation time was hired than the expected one, we were having potential performance regressions.

cc/ @colinsurprenant

[colinsurprenant]

Agree we need some better test harness to avoid future regressions in this area which bit us too much already. I am not sure that adding tests using hardcoded/expected timings is the best approach though; experience has shown that such tests will fail as soon as there is fluctuation in the testing environment. I would suggest two possible alternatives:

• identify what exactly caused such a slowdown and test for that root cause changes; for example: if calling the Janino compiler on a class source is what significantly affect overall startup time and all recent problems where related to ending up compiling too much classes (broken caching) then we could test on the expected number of compiler invocations for a given config in multiple scenarios (# of pipelines / # of workers) and look for regressions on that. It does no rely on hardcoded timing.
• re-re-re setup daily performance benchmarking on stable hardware (for different metrics like: LS startup time, PQ, empty pipeline, complex pipeline, specific plugins, etc) where regressions will be graphed and easily spotted across daily snapshots on master and supported branches. This will provide an even broader perspective of possible performance regressions.