[Transform] Telemetry, Metrics, and Profiling

[prwhelan]

Description

We should look into making Transform performance issues easier to debug using the existing tools supported in Kibana/Elasticsearch. The goal is that users can leverage the existing tools to investigate issues.

Look into:

• Telemetry, both metrics and tracing
• Universal Profiling
• A way for a user to generate the Transform's Search requests so that they can paste it into the dev tools (or make it clearer that enabling debug shows this info)

Already have a preliminary metric for "overhead" and "cardinality" running locally, as well as forwarding the initial tracer when the Transform is started.

### Tasks
- [ ] [Transform] Create hook in AsyncTwoPhase for start/stop checkpoints
- [ ] [Transform] Additional metrics: last*, maxPageSearchSize used for the checkpoint

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[prwhelan]

Notes:

• The existing requests that Transform sends will show up in tracing/metrics under the "unknown" parent, and they're discrete units. It looks like someone is randomly opening PIT, and separately someone else is sending search requests. We can probably create a parent span that says "transforms" or maybe the transform id.
• Look to see if there's already a way to differentiate the user sending the request, then we can set the transform id as the user, that way users with SLO will see just one "transforms" span that can be further broken down by the 10+ SLO ids (for example)
• withScope is opened and closed within a try-catch. This doesn't work for Transforms because the trigger would open it, kick off the Transform task on the generic threadpool, then close it. We'd have to do something like storing it in an AtomicReference or within ThreadLocal and then close it during afterFinishOrFailure. For now, I say we just omit withScope? Likely a question to ask in es-core-infra
• Metrics always seem to record the same value? Or maybe Transforms is just that consistent? The Transform overhead (for my simple dice transform) is always ~19302 nanoseconds.
• OTel recommends having consistent units across an application. Elasticsearch only records Histograms at milliseconds and microseconds. Should we be introducing nanoseconds?
• We need to figure out how this feature would be enabled. Is it always included if telemetry is enabled? Do we have it as a dynamic property in a Transform setting? Do we automatically include it as part of the periodic "audit"? Do we include a /profile API that will spit out the information? Do we include it as part of the /preview, or as an option like /preview?telemetry=true? Etc
• What is the cardinality allowed within OTel/APM? If we add attributes by transform id, there are some clusters with hundreds of transforms, what would be the impact of this?

Decisions:

• Transforms will not introduce a new setting for enabling/disabling tracing. If Elasticsearch has tracing enabled and Transforms has tracing disabled, the user will still see the disparate search/bulk/index/refresh operations that Transforms initiates, so there's really nothing gained by disabling Transform tracing since the traces will be sampled anyway.
• In order to dynamically disable tracing, we'd have to do it within the sampler, which we could follow up on but I'm pretty sure that would be a feature within the APM agent|server.
• To compensate, we should include a label so users can filter out Transforms in their searches. Currently, the Trace will be "internal" from "unknown" type. We can probably include a "transform" label to further filter.

Questions:

1. How much does this cost?
2. Should we include errors initially? It doesn't look like too many people are using errors. If so, how do we correlate them to log messages?