Multi-gauge for metrics at API and Transport level

[sunng87]

What is Multi-gauge

A Multi-gauge is a gauge that allows one or more value fields, each field has its own name. The values in a multi-gauge record share same timestamp, and attributes.

In current spec of OTLP metrics, a gauge can have only one single value. The whole observability ecosystem has been using this protocol for years. However, in a few cases, gauge values are sampled at the same time and share most of their tags. For instance, when we are collecting CPU usage, there are values for different modes: user/sys/io-wait etc. These values are sampled at same time and share most other attributes and timestamp.

Pros

• We will save significant amount of bytes on transport by removing duplicated attributes and timestamps. This can be great improvement since values are mostly integers or floats, but those redundant attributes are strings.
• The new data model will be compatible with single-value gauge if carefully designed.
• Data sinks using table data model, like influxdb, greptimedb and timescaledb, can store multi-gauge values efficiently by treating these fields as table columns. Their query language also has good support for filtering and projection on multi-value gauges.
• Data sinks using Prometheus' single value data structure can still deal with multi-gauge data by transforming it into multiple records that share timestamp and attributes, without any performance lose (and will benefit from reduction of transfer bytes).

Cons

• The ecosystem will need to adopt this new data structure.

Additional context.

• Histogram and Summary can be seen as a special type of multi-gauge.
• Micrometer has built-in support for multi-gauge in its Gauge API. This can be extended to transport as well.
• Influxdb's line protocols allows multiple fields in one record.

[trask]

cc @jmacd I think you've mentioned/thought about this before?

[tigrannajaryan]

@jmacd @lquerel can you please comment on the "wire size" aspect and how the columnar encoding that we already have solves this and whether you see the need to have a different multigauge API?

[lquerel]

Sorry for the long response, but there's a lot to say on this topic.

Multi-gauge is one aspect of what I generally call multivariate metrics (a combination of several metrics, whether they are gauges, counters, histograms, etc.). The lack of native support for multivariate metrics is one of the reasons that motivated me to initiate the OpenTelemetry Protocol with Apache Arrow project in 2021. Initially, the protocol aimed to natively support multivariate metrics, but as mentioned in this issue, ideally, such native support would also require a complete overhaul of the ecosystem, i.e., native support for multivariate metrics in the SDK clients, the protocol, the collector pipeline, and the backends. To achieve a result within a reasonable timeframe, it was decided to focus solely on the protocol part at first. The way the OTel protocol with Apache Arrow aims to address this issue can be summarized by the following steps:

• Step 1 - Unchanged Client SDKs and Backends + Optimized transport: In this step, the OTLP protocol is automatically converted into a columnar representation using the Apache Arrow format (see this project https://github.com/open-telemetry/otel-arrow). Specific processing is carried out to identify and optimize the workarounds generally used by OTel users who actually want to communicate multivariate metrics. For example, the state attribute is used to represent each metric of a multivariate metric (e.g., state=user | sys | io-wait). The columnar representation combined with a sorting mechanism applied at the batch level allows for very efficient compression of such scenarios, at least reducing the network overhead that the type of workaround I just mentioned entails. According to my benchmarks, you can expect to improve the compression rate by a factor of 3 to 7 (the wire-size mentioned by @tigrannajaryan) for this type of telemetry workload containing mainly metrics that are logically multivariate but represented as a collection of univariate metrics. It is therefore now possible to use this protocol to optimize this type of scenario, at least at the transport level. The next phase aims to go further in optimization and native support of this type of metric.
• Step 2 - Client SDKs natively supporting multivariate metrics: To avoid the overhead caused by decomposing multivariate metrics into a collection of univariate metrics at the instrumentation level in applications using this type of signal, native support at the SDK clients' level is necessary. This is also one of the motivations behind the second OTel Weaver project I launched more recently (see https://github.com/open-telemetry/weaver). The idea is to define the concept of Application Telemetry Schema to 1) describe the signals produced by an application (including multivariate metrics), 2) generate optimized SDK clients exposing a "type-safe" API for all the signals described in the schema. So, regarding multivariate metrics, we would have a method that allows collecting all the metrics and attributes of this multivariate metric in a single call. Depending on the configured protocol, we could imagine an optimized sending with a native representation on OTLP or on a protocol using Arrow. Another possible approach is to adapt the existing generic client SDKs to natively report multivariate metrics. However, in my opinion, this would add an additional layer of genericity and thus overheads that were not acceptable for the type of usage that interested me (i.e., instrumentation of systems reporting massive quantities of multivariate metrics).
• Step 3 - Backends natively supporting multivariate metrics: In this phase, the goal is to enable backends to leverage the native representation to completely eliminate the overhead of multivariate metrics end-to-end. There are already certain backends that efficiently support this type of multivariate signals. Otherwise, an automatic conversion into univariate metrics could easily be implemented for backends not supporting them.

To conclude, I believe there must be native and end-to-end support for multivariate metrics. It's not easy because there's an existing ecosystem that needs to be advanced in this direction, but there are efforts underway to improve the situation.

EDIT: added links and comment on wire-size.

[sunng87]

I just heard of otel-arrow project this morning and had a quick look at its readme. The third goal:

Extend OpenTelemetry data model with native support for multi-variate metrics.

is exactly what I want to archive with this issue. And the columnar approach with Arrow format can surely reduce wire-size and improve ingestion speed for the ecosystem.

However, reading the weaver approach, I feel it's a completely overhaul of current OTel metrics data model. I'm afraid we have a long way to go to start multi-variate transform from a totally new wire protocol and upper API. The whole ecosystem can take years to adopt it. (But I still like the idea of strong-typed metrics, the whole downstream ecosystem, dashboarding, alerting, can benefit from it.)

What if we start from a new type like MultiGauge from current data model, API SDK and OTLP? It can be relatively more approachable, and benefit our smooth switch to arrow based transport eventually.

[lquerel]

@sunng87

When I was talking in step 2 about:

Another possible approach is to adapt the existing generic client SDKs to natively report multivariate metrics...

I was referring to an approach that involves extending generic client SDKs and expanding the OTLP protocol with the concept of multivariate metric (multigauge is too restrictive a concept, in my opinion). I didn't follow this approach for various reasons (i.e. performance, SDK usability), but if you or someone else is willing to update the existing client SDKs and OTLP, then I would be pleased to assist with the integration into the OTel with Apache Arrow protocol (referred to as OTAP later).

Note that it's still a long process to achieve because the adaptation of all the client SDKs, the receivers, processors, and exporters is not a small endeavor, and I'm not totally convinced that the approach I'm following will necessarily take much longer. However, as I mentioned, I'm willing to help on the OTAP adaptation and on the specification of the multivariate metric model if this parallel path is retained.

[sunng87]

Thank you for the explanation @lquerel . To me, I'm open to both approaches and willing to do some help just to push the transform forward. I can, for example, seek to add an OTAP-native backend for greptimedb for high performance ingestion.

I would like hear from others of the community about our next steps for this.

[jmacd]

@sunng87 Welcome to the project -- I'm looking forward to future work on Arrow and OpenTelemetry integration and excited to see what you are working on!

I feel that this issue has served its purpose, so I will close it and request specific new issues be filed for some of the tangents discussed here.

The OpenTelemetry Protocol with Apache Arrow project has a way to represent multi-gauge observations, however it is built on a set of assumptions that are not very explicit in our specifications. In the OpenTelemetry metrics data model every data point has a timestamp. In the API requirements there is a specific line that was meant to help us, and we take advantage of it:

The API MUST treat observations from a single Callback as logically taking place at a single instant, such that when recorded, observations from a single callback MUST be reported with identical timestamps.

If the SDK is following this guideline, as I understand it, then every data point written by a callback will be identifiable as being part of a multi-observation, and we should not require any new APIs for asynchronous instruments to emit multiple gauges. On the other hand, we do not have a synchronous API for multiple events in request context, which is something that OpenCensus supported. I've considered the idea of a batch synchronous metrics API but it's only narrowly useful and I believe that there are issues of greater importance for OTel metrics.

Since you mentioned Gauge-Histograms, separately, I would like to refer to some older and closed issues on the topic. I think we'll probably be able to find common ground here. https://github.com/open-telemetry/opentelemetry-proto/issues/274 and https://github.com/open-telemetry/opentelemetry-proto/issues/308.