Open JannikBrand opened 6 months ago
@JannikBrand , @KarstenSchnitter ,
Thank you for creating this issue. If I understand the problem it seems that the Data Prepper OTel metrics source does not accept unknown data. When the input is gRPC, it responds with an error. And when it is HTTP, the data is silently lost. Does that sound correct?
Would you be able to provide a fix for this?
@dlvenable
I only tested sending the unsupported fields via gRPC. Correct, the OTel metrics source does not seem to accept unknown fields (tested via grpcurl) and responds with an error. What I wanted to point out with the OTel collector is the following: When sending the same (previously rejected) payload first to an OTel collector (in my case via HTTP with curl) the collector does not respond with an error. If the OTel collector then continues to export the data to Data Prepper, also Data Prepper accepts the request (possibly due to the collector dropping the fields).
Would you be able to provide a fix for this?
I can also experiment with HTTP instead of gRPC and afterwards look into providing a fix.
Additionally, for traces the flags
field within a span
and within a link
of a span
are not included in the OpenSearch document.
Describe the bug
When ingesting OpenTelemetry metrics into Data Prepper, certain fields from the OTel data model are not supported yet.
When trying to ingest them (e.g. via grpcurl) I get an error returned. Below an example for the scope attributes field:
Here the first aspect from this issue: From the OTLP JSON protobuf encoding doc page I get that the OTLP receiver should not reject those requests.
The second aspect of this issue deals with the thought of adding the unsupported fields. I collected all fields I was aware of:
The following fields will produce an error message like above, hence are unsupported.
attributes
within thescope
field (Link)metadata
within themetrics
(Link)droppedAttributesCount
within thescope
field (Link)min
andmax
fields within thedataPoints
for the histogram / exponential histogram kind (Link Histogram, Link Exponential Histogram)zeroThreshold
field within thedataPoints
for the exponential histogram kind (Link)Next, the fields below do not produce an error, but they are not included in the resulting documents in the DataPrepper sink (stdout / opensearch). It could also be that this is on purpose, but I still wanted to point it out.
droppedAttributesCount
within theresource
field (Link)schemaUrl
within thescopeMetrics
(Link) (Same forscopeLogs,
scopeSpansand
resourceSpans`)schemaUrl
within theresourceMetrics
(Link) which is included in the resulting sink document.To Reproduce Steps to reproduce the behavior:
grpcurl -insecure -d @ < otel-metric-exponential-histogram.json <data_prepper_endpoint>:<data_prepper_metrics_port> opentelemetry.proto.collector.metrics.v1.MetricsService/Export
Note: When I try to send the metric via normal curl to an OTel collector first, which then sends the metric via gRPC to Data Prepper, I do not get the error messages like with grpcurl. However, the fields are also not contained in the resulting documents in the Data Prepper sink. I was using grpcurl to have the direct feedback of Data Prepper without some in-between component like the OTel collector.
Expected behavior
Screenshots If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Additional context Add any other context about the problem here.