Open arend-melissant-tnt opened 1 month ago
receiver/azuremonitor
Sometimes the azure monitor get 0 values, this happens every few minutes
Retrieve azure monitor metric every minute
Some 25-50% of the time 0 values are received, other times >200 values
v0.105
Kubernetes
apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config namespace: {{ tpl .Values.namespace . }} data: config.yaml: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 redis: endpoint: "redis:6379" collection_interval: 60s azuremonitor: subscription_id: ${env:AZURE_SUBSCRIPTION_ID} tenant_id: ${env:AZURE_TENANT_ID} client_id: ${env:AZURE_CLIENT_ID} client_secret: ${env:AZURE_CLIENT_SECRET} resource_groups: - "{{ tpl .Values.resourcegroup . }}" collection_interval: 60s services: - "Microsoft.DBforPostgreSQL/flexibleServers" - "Microsoft.Storage/storageAccounts/blobServices" - "Microsoft.ServiceBus/Namespaces" processors: transform/ignore: error_mode: ignore log_statements: - context: log statements: - set(severity_text, "INFO2") where (severity_text == "INFO" and attributes["channel"] == "messenger") resource/tags: attributes: - key: deployment_environment value: {{ tpl .Values.namespace . }} action: upsert filter/no_info: error_mode: ignore logs: exclude: match_type: strict severity_texts: - "INFO" exporters: otlphttp: endpoint: https://{{ tpl .Values.uptraceurl . }}:443 tls: { insecure: false } headers: uptrace-dsn: 'http://{{ tpl .Values.loggingsecret . }}@{{ tpl .Values.uptraceurl . }}:14318' debug: #verbosity: detailed service: telemetry: logs: level: "info" pipelines: traces: receivers: [otlp] processors: [] exporters: [otlphttp, debug] metrics: receivers: [otlp, redis, azuremonitor] processors: [resource/tags] exporters: [otlphttp, debug] logs: receivers: [otlp] processors: [resource/tags, transform/ignore, filter/no_info] exporters: [otlphttp, debug]
│ 2024-07-26T11:57:51.806Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 115, "data points": 310} ││ 2024-07-26T11:58:48.589Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 0, "metrics": 0, "data points": 0} ││ 2024-07-26T11:58:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T11:59:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T11:59:50.791Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275} ││ 2024-07-26T12:00:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 0, "metrics": 0, "data points": 0} ││ 2024-07-26T12:00:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:01:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:01:50.401Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 145, "data points": 315} ││ 2024-07-26T12:02:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 0, "metrics": 0, "data points": 0} ││ 2024-07-26T12:02:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:03:48.589Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:03:50.091Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275} ││ 2024-07-26T12:04:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:04:49.969Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275} ││ 2024-07-26T12:05:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:05:50.228Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 65, "data points": 65} ││ 2024-07-26T12:06:48.591Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:06:49.848Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 40, "data points": 210} ││ 2024-07-26T12:07:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:07:50.399Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 75, "data points": 75} ││ 2024-07-26T12:08:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:08:49.876Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 40, "data points": 210} ││ 2024-07-26T12:09:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:09:49.913Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275} ││ 2024-07-26T12:10:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:10:50.022Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 95, "data points": 265} ││ 2024-07-26T12:11:48.589Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:11:49.068Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 10, "data points": 10} ││ 2024-07-26T12:12:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:12:50.173Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 95, "data points": 265} ││ 2024-07-26T12:13:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:13:50.013Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 75, "data points": 75} ││ 2024-07-26T12:14:48.590Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32} ││ 2024-07-26T12:14:49.761Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 40, "data points": 210} ││ 2024-07-26T12:15:48.589Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}
No response
Pinging code owners:
receiver/azuremonitor: @nslaughter @codeboten
See Adding Labels via Comments if you do not have permissions to add labels yourself.
Component(s)
receiver/azuremonitor
What happened?
Description
Sometimes the azure monitor get 0 values, this happens every few minutes
Steps to Reproduce
Expected Result
Retrieve azure monitor metric every minute
Actual Result
Some 25-50% of the time 0 values are received, other times >200 values
Collector version
v0.105
Environment information
Environment
Kubernetes
OpenTelemetry Collector configuration
Log output
Additional context
No response