[BUG] Otel / data-prepper 404 error with metrics pipelines

[jinfanzon]

Describe the bug When metrics are exported from OTEL to data-prepper the following 404 error es received:

0tel-collector | 2022-08-12T20:18:18.517Z error exporterhelper/queued_retry.go:183 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlp/http", "error": "Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 173}

To Reproduce Steps to reproduce the behavior:

1. I launch data-prepper with OpenSearch as a sink for thre pipelines: "raw-pipeline" ,"service-map-pipeline" and "metrics-pipeline":
2. I launch Oel with the following config:

For metrics, I've tried the exporters otlp, otlp/http, otlp/2 and otlp/data-prepper. All unsuccesfully.

3. I instrument a Java Application

Expected behavior Metrics traces sould be imported into OpenSearch. Just to mention, the traces pipeline works OK

Environment (please complete the following information):

• OS: Ubuntu 20.04 LTS
• otel/opentelemetry-collector-contrib:latest
• opensearchproject/data-prepper:latest

[dinujoh]

@jinfanzon Thanks for your interest in Data Prepper!

I see in the data prepper configuration for otel_metrics_source, port was not specified. When port is not specified, the default port is 21891.

Can you update the OTel exporter configuration to use the endpoint: data-prepper:21891 and retry running OTel exporter ?

[jinfanzon]

Thanks @dinujoh - I have the following issue, as you may see both traces and service-map are exported otlp/2 (http). after checking the documentation you provided for metrics, it only uses otlp/grpc, so I've modified the otel configuration adding a otlp/grpc exporter:

And when I run data-prepper I get the following error Invalid record type. OpenSearch sink only supports String and Events:

data-prepper | 2022-08-16T13:36:48,975 [otel-metrics-pipeline-sink-worker-2-thread-1] ERROR com.amazon.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [otel-metrics-pipeline] process worker encountered a fatal exception, cannot proceed further data-prepper | java.util.concurrent.ExecutionException: java.lang.RuntimeException: Invalid record type. OpenSearch sink only supports String and Events data-prepper | at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?] data-prepper | at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?] data-prepper | at com.amazon.dataprepper.pipeline.common.PipelineThreadPoolExecutor.afterExecute(PipelineThreadPoolExecutor.java:70) ~[data-prepper.jar:1.5.1] data-prepper | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131) ~[?:?] data-prepper | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?] data-prepper | at java.lang.Thread.run(Thread.java:832) ~[?:?] data-prepper | Caused by: java.lang.RuntimeException: Invalid record type. OpenSearch sink only supports String and Events data-prepper | at com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink.getDocument(OpenSearchSink.java:178) ~[data-prepper.jar:1.5.1] data-prepper | at com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink.doOutput(OpenSearchSink.java:130) ~[data-prepper.jar:1.5.1] data-prepper | at com.amazon.dataprepper.model.sink.AbstractSink.lambda$output$0(AbstractSink.java:38) ~[data-prepper.jar:1.5.1] data-prepper | at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:89) ~[data-prepper.jar:1.5.1] data-prepper | at com.amazon.dataprepper.model.sink.AbstractSink.output(AbstractSink.java:38) ~[data-prepper.jar:1.5.1] data-prepper | at com.amazon.dataprepper.pipeline.Pipeline.lambda$publishToSinks$4(Pipeline.java:231) ~[data-prepper.jar:1.5.1] data-prepper | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] data-prepper | at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] data-prepper | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?] data-prepper | ... 2 more

[asifsmohammed]

Thanks @jinfanzon for updating the configuration. While we try to reproduce this issue, can you try out stdout sink and check what data you see. Ip possible can you share an example metrics data of how it looks?

We noticed that the otel-metrics-pipeline has metrics-pipeline pipeline as a sink. And from the traceback provided error was caused by otel-metrics-pipeline which doesn't have OpenSearch sink.

[jinfanzon]

Hi @asifsmohammed - I was able to send metrics to sink stdout. Please check my pastebin

This is my config:

Data-Prepper:

Otel:

So as soon as I change to sink to opensearch I get again the issue:

Please be aware that the log was longer, I just added a part of it

[jinfanzon]

Hi - I was missing config:

` processor:

• otel_metrics_raw_processor:`

I've also added a required field: index.

I'm wondering if there is any documentation of how to visualize the processed metrics in OpenDashboard

[cmanning09]

We do not have any special visualization plugins specifically tailored for OTEL metrics. This is something I believe the OpenSearch Dashboards project is working on as we do want to support traces, logs and metrics in the dashboards. @anirudha maybe able to provide more clarity here.