Document online profiling setup and process

[allenlsy]

Please describe the Improvement and/or Feature Request

Support adding a continuous profiling tool helps to monitor the performance of OSM in more detail than the current observability tools, as it periodically runs the profiling process and stores stack traces to a centralized server. There are existing solutions for this purpose. Some options are:

• Pyroscope: https://github.com/pyroscope-io/pyroscope
• Profefe: https://github.com/profefe/profefe

~To support continuous profiling, we can add a toggle switch in MeshConfig, in the observability section, to allow users installing a tool in the cluster.~

As discussed in https://github.com/openservicemesh/osm/pull/4698, Pyroscope will not be installed via OSM directly. We will provide a document instructed by the Pyroscope documentation to install it inside a running mesh.

Root tracking issue: #4622

Scope (please mark with X where applicable)

• New Functionality [X]
• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [ ]
• Egress [ ]
• Envoy Control Plane [ ]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [ ]
• Logging [ ]
• Debugging [X]
• Tests [ ]
• CI System [ ]
• Demo [ ]
• Project Release [ ]

Possible use cases

• When there is a resource consumption spike, with continuous profiling, we can check at which line of code causes the spike. This is not possible with only Prometheus.
• Performance optimization research requires this tool.

[allenlsy]

Issue completed