irfansharif
commented
2 years ago Another possible benefit of integrating with prometheus exemplars: abusing it to build machinery for per-statement/replica/
- Defining a finite number of prometheus labels to export per-statement metrics over;
- CRDB choosing what the most important statements are to collect metrics over and ensure it's able to change that set over time (but not too frequently);
- For each statement in that set, map it to one of the finite prometheus labels and export data through it;
- Whenever set membership changes, emit some annotation with metadata (this would be the exemplar event) such that you can look at a single timeseries and know that from time T to time T+N, it represented statement S.
github-actions[bot]
Is your feature request related to a problem? Please describe.
It would be very useful to be able to go from an interesting event/point-in-time as observed by metrics to a corresponding trace. Grafana for e.g. has an in-built trace-view now (replacing jaeger/zipkin), and with a bit of integration, lets you jump from an exemplar event as observed in your metrics charts to a corresponding trace. This article has more details and examples.
Describe the solution you'd like
Our tracing libraries are already OpenTelemetry-compatible (which is what makes them viewable through jaeger). And we already export our metrics through the prometheus format, so I assume there are 3rd-party libraries we can integrate with to export them in the OpenMetrics format that this exemplar machinery seems to want. What's left to do here perhaps is connecting some dots.
Jira issue: CRDB-16907
Epic CRDB-32402