Closed abkhan5 closed 6 months ago
Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.
Pinging code owners for exporter/prometheus: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself.
Looks like this is actually using the otelhttp exporter?
Looks like this is actually using the otelhttp exporter?
i've tried all three of the exporters mentioned . Each give different error messages The promethues exporter gives the following
no existing monitoring routine is r │ │ 2024/03/23 01:39:11 collector server run finished with error: cannot start pipelines: listen tcp 10.0.117.182:80: bind: cannot assign requested address;
the oltphttp/prometheus exporter gives this error exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/prometheus", "error": "not retryable error: Permanent error: error exporting items, request to http://prometheus-server.monitoring.svc.cluster.local:80/v1/metrics responded with HTTP Status Code 404", "dropped_items": 107}
and remotewrite gives
exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination for metric "system.disk.io"; invalid temporality and type combination for metric "system.disk.io_time"; invalid temporality and type combination for metric "system.disk.merged"; invalid temporality and type combination for metric "system.disk.operation_time"; invalid temporality and type combination for metric "system.disk.operations"; invalid temporality and type combination for metric "system.disk.weighted_io_time"; invalid temporality and type combination for metric "system.cpu.time"; Permanent error: Permanent error: context deadline exceeded", "errorCauses": [{"error": "Permanent error: invalid temporality and type combination for metric "system.disk.io"; invalid temporality and type combination for metric "system.disk.io_time"; invalid temporality and type combination for metric "system.disk.merged"; invalid temporality and type combination for metric "system.disk.operation_time"; invalid temporality and type combination for metric "system.disk.operations"; invalid temporality and type combination for metric "system.disk.weighted_io_time"; invalid temporality and type combination for metric "system.cpu.time""}, {"error": "Permanent error: Permanent error: context deadline exceeded"}], "dropped_items": 107}
To send OTLP to prometheus, you need to enable OTLP ingestion on the prometheus server: https://prometheus.io/docs/prometheus/latest/feature_flags/#otlp-receiver
To send any metrics to Prometheus today, you need to make sure they aggregation temporality is Cumulative, not delta. The errors in https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/31914#issuecomment-2016294872 indicate you are trying to send Delta metrics.
The prometheus exporter exposes a local endpoint on the collector (e.g. localhost:8080) which a prometheus server can scrape. It doesn't really make sense to listen on a remote IP.
Opentelemetry exporter on Azure Kubernetes on trying to connect to Prometheus throws an error
OpenTelemetry exporter on being configured to connect to Prometheus server on the same cluster throws the following error _024-03-22T11:50:22.849Z error exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/prometheus", "error": "not retryable error: Permanent error: error exporting items, request to http://prometheus-server.monitoring.svc.cluster.local:80/v1/metrics responded with HTTP Status Code 404", "droppeditems": 107}
Install Prometheus on Azure Kubernetes on the monitoring namespace using helm chart
helm install prometheus prometheus-community/prometheus --debug -n monitoring
On installing prometheues this is what the services look like
Install opentelemetry using helm with a values.yaml file which looks like the code below
helm install opentelemetry-collector open-telemetry/opentelemetry-collector --debug -f values.yaml;
The values.yaml file looks like this
I expected apps deployed on AKS to show up in Prometheus . I also expected the logs in the opentelemetry pod to show it successfully able to connect and for metrics to show up in Prometheus
What I see instead are 404 error on trying to connect to prometheus . The errors look like this
error exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/prometheus", "error": "not retryable error: Permanent error: error exporting items, request to http://prometheus-server.monitoring.svc.cluster.local:80/v1/metrics responded with HTTP Status Code 404", "dropped_items": 107}
While trying the prometheusremotewrite expoerter i get the following error
exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination for metric \"system.disk.io\"; invalid temporality and type combination for metric \"system.disk.io_time\"; invalid temporality and type combination for metric \"system.disk.merged\"; invalid temporality and type combination for metric \"system.disk.operation_time\"; invalid temporality and type combination for metric \"system.disk.operations\"; invalid temporality and type combination for metric \"system.disk.weighted_io_time\"; invalid temporality and type combination for metric \"system.cpu.time\"; Permanent error: Permanent error: context deadline exceeded", "errorCauses": [{"error": "Permanent error: invalid temporality and type combination for metric \"system.disk.io\"; invalid temporality and type combination for metric \"system.disk.io_time\"; invalid temporality and type combination for metric \"system.disk.merged\"; invalid temporality and type combination for metric \"system.disk.operation_time\"; invalid temporality and type combination for metric \"system.disk.operations\"; invalid temporality and type combination for metric \"system.disk.weighted_io_time\"; invalid temporality and type combination for metric \"system.cpu.time\""}, {"error": "Permanent error: Permanent error: context deadline exceeded"}], "dropped_items": 107}