[BUG] Unsupported OTel metric, instrumentation scope and resource fields

[JannikBrand]

Describe the bug

When ingesting OpenTelemetry metrics into Data Prepper, certain fields from the OTel data model are not supported yet.

When trying to ingest them (e.g. via grpcurl) I get an error returned. Below an example for the scope attributes field:

Error invoking method "opentelemetry.proto.collector.metrics.v1.MetricsService/Export": error getting request data: message type opentelemetry.proto.common.v1.InstrumentationScope has no known field named attributes

Here the first aspect from this issue: From the OTLP JSON protobuf encoding doc page I get that the OTLP receiver should not reject those requests.

OTLP/JSON receivers MUST ignore message fields with unknown names and MUST unmarshal the message as if the unknown field was not present in the payload. This aligns with the behavior of the Binary Protobuf unmarshaler and ensures that adding new fields to OTLP messages does not break existing receivers.

---

The second aspect of this issue deals with the thought of adding the unsupported fields. I collected all fields I was aware of:

The following fields will produce an error message like above, hence are unsupported.

• attributes within the scope field (Link)
• metadata within the metrics (Link)
• droppedAttributesCount within the scope field (Link)
• the min and max fields within the dataPoints for the histogram / exponential histogram kind (Link Histogram, Link Exponential Histogram)
• the zeroThreshold field within the dataPoints for the exponential histogram kind (Link)

Next, the fields below do not produce an error, but they are not included in the resulting documents in the DataPrepper sink (stdout / opensearch). It could also be that this is on purpose, but I still wanted to point it out.

• droppedAttributesCount within the resource field (Link)
• schemaUrl within the scopeMetrics (Link) (Same for scopeLogs,scopeSpansandresourceSpans`)
  • There is also the schemaUrl within the resourceMetrics (Link) which is included in the resulting sink document.

---

To Reproduce Steps to reproduce the behavior:

• Run Data Prepper with an OTel metrics pipeline. Here an example configuration file:

metrics-pipeline:
      workers: 4
      source:
        otel_metrics_source:
          proto_reflection_service: true
          ssl: false
      buffer:
        bounded_blocking:
          buffer_size: 512
          batch_size: 8
      processor:
        - otel_metrics:
      sink:
        - stdout:

• Prepare the JSON payload. For example here for an exponential histogram kind (I am choosing this kind because it can contain all the mentioned unsupported fields). I just put some example numbers and did not think too much about if the values make sense. The main goal was to have a complete payload specifying every field there is in the OpenTelemetry Proto Specifications.

{
    "resourceMetrics": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "service.name",
                        "value": {
                            "stringValue": "basic-metric-service"
                        }
                    }
                ],
                "droppedAttributesCount": 5
            },
            "scopeMetrics": [
                {
                    "metrics": [
                        {
                            "description": "Example of n exponential histogram",
                            "name": "requests",
                            "exponentialHistogram": {
                                "aggregationTemporality": 2,
                                "dataPoints": [
                                    {
                                        "attributes": [
                                            {
                                                "key": "datapoint.myattribute",
                                                "value": {
                                                    "stringValue": "metricAttribute"
                                                }
                                            }
                                        ],
                                        "startTimeUnixNano": 1660736598000000000,
                                        "timeUnixNano": 1660736598000001000,
                                        "count": 4,
                                        "sum": 5,
                                        "scale": 5,
                                        "zeroCount": 2,
                                        "positive": {
                                            "offset": 3,
                                            "bucketCounts": [
                                                2,
                                                2
                                            ]
                                        },
                                        "negative":{
                                            "offset": 3,
                                            "bucketCounts": [
                                                2,
                                                2
                                            ]
                                        },
                                        "exemplars": [
                                            {
                                                "filteredAttributes": [
                                                    {
                                                        "key": "datapoint.exemplarattribute",
                                                        "value": {
                                                            "stringValue": "exemplarAttributeValue"
                                                        }
                                                    }
                                                ],
                                                "timeUnixNano": 1660736598000001000,
                                                "asDouble": 1,
                                                "traceId": "428264014a59a9a29b7053279f687e9f",
                                                "spanId": "9bc01dfad9f631ff"
                                            }
                                        ],
                                        "flags": 1,
                                        "min": 7.6,
                                        "max": 9.6,
                                        "zeroThreshold": 99   
                                    }
                                ]
                            },
                            "unit": "1",
                            "metadata": [
                                {
                                    "key": "myMetadata",
                                    "value": {
                                        "stringValue": "exampleValue"
                                    }
                                }
                            ]
                        }
                    ],
                    "scope": {
                        "name": "example-exporter-collector",
                        "version": "MyVersion",
                        "attributes": [
                            {
                              "key": "myScopeAttribute",
                              "value": { "stringValue": "my-scope-attribute-value" }
                            }
                        ],
                        "droppedAttributesCount": 1
                    },
                    "schemaUrl": "https://opentelemetry.io/schemas/1.24.0"
                }
            ],
            "schemaUrl": "https://opentelemetry.io/schemas/1.24.0"
        }
    ]
}

• Send this data to your Data Prepper endpoint.
  • e.g. using grpcurl: grpcurl -insecure -d @ < otel-metric-exponential-histogram.json <data_prepper_endpoint>:<data_prepper_metrics_port> opentelemetry.proto.collector.metrics.v1.MetricsService/Export
• There will be errors like the one mentioned in the beginning. You can remove the fields which are producing the error from the payload until it works. Then you will get something like the following document in the stdout sink of Data Prepper. It misses the two fields mentioned in the second list above.

  {"kind":"SUM","flags":1,"description":"Example of a Counter","serviceName":"basic-metric-service","schemaUrl":"https://opentelemetry.io/schemas/1.24.0","isMonotonic":true,"unit":"1","aggregationTemporality":"AGGREGATION_TEMPORALITY_CUMULATIVE","exemplars":[{"time":"2022-08-17T11:43:18.000001Z","value":1.0,"attributes":{"exemplar.attributes.datapoint@exemplarattribute":"exemplarAttributeValue"},"spanId":"f5b734d5d7da77d7fadf57df","traceId":"e36f36eb8d35e1ae7d6bd6b6f5bef4e77dbbf5febcedef5f"}],"name":"requests","startTime":"2022-08-17T11:43:18Z","time":"2022-08-17T11:43:18.000001Z","value":1.0,"instrumentationScope.name":"example-exporter-collector","metric.attributes.datapoint@myattribute":"metricAttribute","resource.attributes.service@name":"basic-metric-service", "instrumentationScope.version":"MyVersion"}

Note: When I try to send the metric via normal curl to an OTel collector first, which then sends the metric via gRPC to Data Prepper, I do not get the error messages like with grpcurl. However, the fields are also not contained in the resulting documents in the Data Prepper sink. I was using grpcurl to have the direct feedback of Data Prepper without some in-between component like the OTel collector.

Expected behavior

• If there are unsupported fields, the request should not fail.
• Unsupported fields like the scope attributes should also be included in the resulting documents.

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Data Prepper 2.6.1
• grpcurl 1.8.9

Additional context Add any other context about the problem here.

[dlvenable]

@JannikBrand , @KarstenSchnitter ,

Thank you for creating this issue. If I understand the problem it seems that the Data Prepper OTel metrics source does not accept unknown data. When the input is gRPC, it responds with an error. And when it is HTTP, the data is silently lost. Does that sound correct?

Would you be able to provide a fix for this?

[JannikBrand]

@dlvenable

I only tested sending the unsupported fields via gRPC. Correct, the OTel metrics source does not seem to accept unknown fields (tested via grpcurl) and responds with an error. What I wanted to point out with the OTel collector is the following: When sending the same (previously rejected) payload first to an OTel collector (in my case via HTTP with curl) the collector does not respond with an error. If the OTel collector then continues to export the data to Data Prepper, also Data Prepper accepts the request (possibly due to the collector dropping the fields).

Would you be able to provide a fix for this?

I can also experiment with HTTP instead of gRPC and afterwards look into providing a fix.

[JannikBrand]

Additionally, for traces the flags field within a span and within a link of a span are not included in the OpenSearch document.