JaredTan95
commented
1 month ago you need move this issue into https://github.com/Bhoopesh123/OpenTelemetry repo first, It looks like there is a configuration error. The address of the otlp exportaer was incorrectly configured for prometheus; however, prometheus is not able to receive otlp metrics directly. https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations
Component(s)
No response
Describe the issue you're reporting
I used https://github.com/Bhoopesh123/OpenTelemetry for setting up Open telemetry on my EKS. Only prometheus metrics are failing. Im able to check the traces in jaeger.
info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 20, "metrics": 40, "data points": 180} 2024-05-22T11:23:55.757Z error exporterhelper/common.go:95 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/prometheus", "error": "not retryable error: Permanent error: error exporting items, request to http://opentelemetry-demo-prometheus-server:9090/api/v1/otlp/v1/metrics responded with HTTP Status Code 404", "dropped_items": 180} go.opentelemetry.io/collector/exporter/otlphttpexporter.createMetricsExporter.WithQueue.func5.1
Prometheus server is running banandh@MCHICL652W60 opentelemetry-app % k logs opentelemetry-demo-prometheus-server-59bd64f76-w8tg6 -n otel-demo ts=2024-05-21T05:33:37.140Z caller=main.go:178 level=info msg="Experimental in-memory exemplar storage enabled" ts=2024-05-21T05:33:37.779Z caller=main.go:585 level=info msg="Starting Prometheus Server" mode=server version="(version=2.46.0, branch=HEAD, revision=cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)" ts=2024-05-21T05:33:37.794Z caller=main.go:590 level=info build_context="(go=go1.20.6, platform=linux/arm64, user=root@42454fc0f41e, date=20230725-12:37:33, tags=netgo,builtinassets,stringlabels)" ts=2024-05-21T05:33:37.798Z caller=main.go:591 level=info host_details="(Linux 6.6.14-0-virt https://github.com/Bhoopesh123/OpenTelemetry/issues/1-Alpine SMP Fri, 26 Jan 2024 11:08:07 +0000 aarch64 opentelemetry-demo-prometheus-server-59bd64f76-w8tg6 (none))" ts=2024-05-21T05:33:37.798Z caller=main.go:592 level=info fd_limits="(soft=1048576, hard=1048576)" ts=2024-05-21T05:33:37.798Z caller=main.go:593 level=info vm_limits="(soft=unlimited, hard=unlimited)" ts=2024-05-21T05:33:37.934Z caller=web.go:563 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 ts=2024-05-21T05:33:37.982Z caller=main.go:1026 level=info msg="Starting TSDB ..."
Im able to access prometheus in localhost:9090 after port-forward. Pls help