Baseline support for otelpyroscope

[petethepig]

Context

We have an otel integration located here. It allows users to integrate with tracing providers and store profiles for each span in pyroscope.

Goal

We want to have ability to go from an individual profile (e.g app{profile_id="abcdef"}) to it's "baseline", which would an aggregation of traces in the same execution context.

Proposed Solution

Now that we have support for comparing queries (https://github.com/pyroscope-io/pyroscope/pull/942), the easiest way to implement this would be an extension of otelpyroscope, where instead of (or in additon to) posting the url to an individual profile (e.g http://localhost:4040/render?query=app.cpu{profile_id="abcdef"}, we would post a url to comparison view, e.g http://localhost:4040/render?leftQuery=app.cpu{controller="foo"}&rightQuery=app.cpu{profile_id="abcdef"}

[Rperry2174]

I think this should be "in addition too" so in the end there should be two fields:

1. single view (which links directly to current profile_id)
2. comparson view (which has one "side" being the profile_id and the other side being all of the other tags available in the context (minus profile_id)

Also this relies on #942 which we merged recently

[kolesnikovae]

I'll also add an option to add the current root span name to labels automatically - for those who want one-to-one mapping between a trace and profile but don't want to mess with pprof labels.

[Rperry2174]

So just to summarize here so we're on the same page... The goal is to make it so that we can easily compare a single trace-specific profile to a baseline of an aggregation of all the "similar" trace-specific-profiles

To illustrate we can use our golang-push example there are two tags:

• vehicle: car, bike, or scooter
• region: us-east-1, us-west-1, eu-west-1

Now when using the otelpyroscope package we would be able to introduce an extra tag which is span_name corresponding with name of the root span. This would make it so that if we have:

• region: us-east-1
• vehicle: car
• trace specific span where span name is CarHandler i.e. (span_name: CarHandler)

Then if the WithAutoLabels option is set to:

• WithAutoLabels=true then we would automatically add the tag/label of span_name and write the following to the database: app.cpu{region='us-east-1', vehicle='car', span_name='CarHandler'}
• WithAutoLabels=false then we would not add the tag/label and instead we would write the following to the database: app.cpu{region='us-east-1', vehicle='car'}

There are a few main benefit and some potential disadvantages to implementing this in otelpyroscope this way:

Benefits:

• Users who want a one-to-one mapping between a root-span and a profile can simply select the span_name='CarHandler' when they would like to use the baseline to compare a profile to the CarHandler span
• Users can do this without having to mess with labels in their code

  (ignore coloring small issue to be fixed later)

Disadvantages:

• if for some reason two spans that are not actually the same share the same name it will pollute the baseline - Hopefully it is fair to assume people would not do this (?) ^^^
• if for some reason people have a dynamically named span_name it will introduce unnecessary and potentially unusable cardinality to the data set (i.e. if app.cpu{span_name: 'CarHandler-293949'} ) - Hopefully it is fair to assume people will not have dynamically generated span names (?) ^^^

[petethepig]

And after #967 we'll have diff support too.

[petethepig]

Implemented in https://github.com/pyroscope-io/otelpyroscope/pull/1