[Feature] Profiling Support during workload run

[sandeshkr419]

Is your feature request related to a problem? Please describe.

CPU profiling helps in essential development tasks to monitor resource usage, improve algorithm efficiency. Suggesting integrating async-profiler as part of OSB runs.

This project is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on HotSpot JVM.

Describe the solution you'd like

Since profiler is to run on a specific node only, the usage can be restricted to clusters which are hosted locally, i.e., localhost:9200. This omits the complexity to fetching node details when a search endpoint is provided instead.

As an introductory feature, the profiling can be restricted to obtain a flamegraph (.html) output.

In OSB run command, --profiling=flamegraph can be provided as argument to indicate if profiling is required.

As part of implementation, the profiler can be downloaded (if not downloaded), and the the profiler can be started with start option before starting any other cluster indexing/search activity and then stop option can be used at the end (refer async-profiler doc). The final OSB output can display the path to .html profiling output that is generated.

Additional context

This will assist developers working on improving indexing/search performance in OpenSearch.

Rally support for telemetry: https://esrally.readthedocs.io/en/stable/telemetry.html#jfr Not sure if we can leverage these as well.

Welcome suggestions!

[gkamat]

This is a needed capability. The exact design should be scoped out before implementation can begin.