Closed cobar79 closed 7 months ago
/cc @jpkrohling
According to Mimir http api it is posting to the correct Distributor endpoint. Maybe it is in the Load Balancer.
After reviewing prometheus config which contained a remote write with a header. I added header to the collector otlphttp exporter
otlphttp:
endpoint: http://localhost:9009/otlp
headers:
X-Scope-OrgID: osint
Still connection refused. prometheusremotewrite still provides no feed back and no results.
@aknuds1 , @krajorama , would you be able to take a look at this one?
Taking a step back, I figured it out. Container to Container communications is via service name. Changed to load-balancer service name
otlphttp:
endpoint: http://load-balancer:9009/otlp
headers:
X-Scope-OrgID: osint
Describe the bug Apologies if this not the correct place, but I can't find any examples pushing metrics using OTEL collector and Mimir backend. Problem set is pushing metrics from Spring Boot 3 to Collector and exporting to Mimir backend is failing for connection refused. It appears the OTLP URI is not being changed when attempting to export. Traces and Logs are exporting correctly.
Following Grafana Mimir collector configuration First attempt to export using "prometheusremotewrite" export showed no errors and nothing was exported. Second attempt to export to "otlphttp" endpoint revealed a
Noting the endpoint is the receiver value "http://localhost:4318/v1/metrics" via spring boot actuator value. The export should have been "http://localhost:9009/otlp" not "http://localhost:9009/otlp/v1/metrics"
Do I need to do something in the processor to remove the v1/metrics?
Environment: Spring Boot 3.2.1 Docker 4.15.0 Patterned after Play With Mirmir and adapted to OTEL Collector.
Steps to reproduce
What did you expect to see? Metrics showing up in Dashboard. Metrics persisted in Minio. Prometheus (Mimir) explore to contain metrics to search. Prometheus should be scraping Minio DB.
What did you see instead? Nothing in minio database console. Nothing in Mimir Write dashboard. Nothing in the Mimir explore dropdowns.
What version did you use?
What config did you use? compose:
Prometheus:
Grafana Datasource:
Collector:
Spring Boot Microservice
Load Balancer:
Environment Spring Boot 3 JDK 1.17 opentelemetry-collector-contrib:0.91.0 Docker: 4.15.0 (93002)
Additional context