How to model Prometheus Gauge in OpenTelemetry language?

[moorara]

Use-Case

I want to have an instrument similar to Prometheus Gauge metric to measure the number of in-flight request. Basically, on each request I need to increase and once the request is handled, decrease it. The aggregation is also should be last value I believe.

Any suggestion how can I approach this use-case?

[moorara]

I can see OpenTelemetry spec has defined an UpDownCounter instrument, but I cannot figure out how to implement it using OpenTelemetry Go SDK.

[jmacd]

The 0.4 specification has not been formally released yet, and the language SDKs are not caught up yet. #709 is the key next step toward adding the new instruments, after which I plan to add multi-observer support (see #634), followed by the new instruments.

[jmacd]

I believe this will be resolved: https://github.com/open-telemetry/opentelemetry-go/issues/708

[MrAlias]

@jmacd did you mean https://github.com/open-telemetry/oteps/pull/118?

[moorara]

I was able to use the new UpDownCounter in 0.6.0 release and implement a gauge, so I close this.

[otherview]

Hey @moorara any chance you can show a snippet of the code ? I'm also stuck trying to create a Gauge too.

Thanks!

[moorara]

@otherview

This is a simple example:

import "go.opentelemetry.io/otel/api/metric"

type instruments struct {
    reqCounter  metric.Int64Counter
    reqGauge    metric.Int64UpDownCounter
}

func newInstruments(meter metric.Meter) *instruments {
    mm := metric.Must(meter)

    return &instruments{
        reqCounter: mm.NewInt64Counter(
            "http_requests_total",
            metric.WithDescription("The total number of http requests"),
            metric.WithUnit(unit.Dimensionless),
        ),
        reqGauge: mm.NewInt64UpDownCounter(
            "http_requests_active",
            metric.WithDescription("The number of in-flight http requests"),
            metric.WithUnit(unit.Dimensionless),
        ),
    }
}

And here is how you can measure the metrics:

...
m.instruments.reqGauge.Add(ctx, 1,
    label.String("method", "GET"),
    label.String("route", "/"),
)
...
m.instruments.reqGauge.Add(ctx, -1,
    label.String("method", "GET"),
    label.String("route", "/"),
)
...

You can see the full code here.

[otherview]

Thanks for the swift reply @moorara !

So.. to replicate a promauto.NewGauge().Set() (prometheus client API) I would have to store and subtract the existing value in otel?

[seanhoughton]

I would like to ask that this issue be re-opened because the answer doesn't work for a large number of use cases where the instrumentation doesn't have access to each value increment/decrement events. It might work for the specific case of an http handler, but not for many classes of instrumentation where change event instrumentation is not possible. Examples include water levels, engine temperatures, disk consumption, and number of processes running.

The opentelemetry-collector's host metrics is a good example of where this is needed and that codebase goes through some gymnastics to implements gauge types internally, although the types are private. Example: https://github.com/open-telemetry/opentelemetry-collector/blob/08a18d5b9f1b8ee001f8e16e6f6b9daf45b4d42e/cmd/mdatagen/metricdata.go#L104

The fact that a tool written by the opentelemetry team implemented a full suite of gauges in one of its own project indicates to me that they should just be implemented in the core library as first class instruments or at least promoted to a public util library for use in other projects.

[seanhoughton]

Ultimately the solution looks like something that is still to be worked on. Instead of using a dedicated instrument like I mentioned above you would use a ValueObserver/ValueRecorder and make sure the aggregation selector uses the last value instead of producing a histogram (which is baked in to the prometheus convenience functions). The workaround is to make your own selector which will let you return a "lastvalue" aggregator for value recorders and observers.

See: https://github.com/open-telemetry/opentelemetry-go/issues/1139#issuecomment-690774659