search

open-telemetry / opentelemetry-collector

OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
4.36k stars 1.44k forks source link

Prometheus receiver crashes on startup #2705

Closed DanTulovsky closed 3 years ago

DanTulovsky commented 3 years ago

Describe the bug otle-collector crashes on startup

Steps to reproduce If possible, provide a recipe for reproducing the error.

What did you expect to see? No crash on startup :)

What did you see instead?

2021-03-15T15:04:06.396Z info service/service.go:411 Starting OpenTelemetry Contrib Collector... {"Version": "v0.19.0", "GitHash": "63b2f339", "NumCPU": 2}
2021-03-15T15:04:06.396Z info service/service.go:255 Setting up own telemetry...
2021-03-15T15:04:06.398Z info service/telemetry.go:102 Serving Prometheus metrics {"address": ":8888", "level": 0, "service.instance.id": "173bdd06-d3bc-4903-ba90-eff33a3cc621"}
2021-03-15T15:04:06.399Z info service/service.go:292 Loading configuration...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0x88a85c]

goroutine 1 [running]:
github.com/spf13/viper.(*Viper).AllKeys(0x0, 0x30, 0x6, 0xc000947301)
/home/circleci/go/pkg/mod/github.com/spf13/viper@v1.7.1/viper.go:1824 +0x9c
github.com/spf13/viper.(*Viper).AllSettings(0x0, 0x36e9b5a)
/home/circleci/go/pkg/mod/github.com/spf13/viper@v1.7.1/viper.go:1904 +0x51
go.opentelemetry.io/collector/receiver/prometheusreceiver.customUnmarshaler(0xc00088bd40, 0x313b880, 0xc000530840, 0xc0009474f0, 0xfbb7cc)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/receiver/prometheusreceiver/factory.go:70 +0x139
go.opentelemetry.io/collector/receiver/receiverhelper.(*factoryWithUnmarshaler).Unmarshal(0xc00012b150, 0xc00088bd40, 0x313b880, 0xc000530840, 0x7f984d43db48, 0xc00012b150)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/receiver/receiverhelper/factory.go:156 +0x4d
go.opentelemetry.io/collector/config.LoadReceiver(0xc00088bd40, 0xc00002baf6, 0xa, 0xc00002baf6, 0xa, 0x3e59100, 0xc00012b150, 0x0, 0xc0002f5560, 0x0, ...)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/config/config.go:319 +0x11f
go.opentelemetry.io/collector/config.loadReceivers(0xc00086fa40, 0xc000569770, 0x9, 0xc00086fa40, 0x0)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/config/config.go:347 +0x1df
go.opentelemetry.io/collector/config.Load(0xc000287b00, 0xc000569770, 0xc000504300, 0xc000569a70, 0xc000569740, 0x52, 0x124, 0x5a3ea88)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/config/config.go:159 +0x258
go.opentelemetry.io/collector/service.FileLoaderConfigFactory(0xc000287b00, 0xc0006df340, 0xc000569770, 0xc000504300, 0xc000569a70, 0xc000569740, 0xc00030ff20, 0x0, 0x0)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/service/service.go:138 +0x25a
go.opentelemetry.io/collector/service.(*Application).setupConfigurationComponents(0xc0002f0180, 0x3e41d80, 0xc000116000, 0x382a9b0, 0x0, 0x3)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/service/service.go:293 +0xf8
go.opentelemetry.io/collector/service.(*Application).execute(0xc0002f0180, 0x3e41d80, 0xc000116000, 0x382a9b0, 0x0, 0x0)
/home/circleci/go/pkg/mod/go.opentelemetry.io/collector@v0.19.0/service/service.go:429 +0x4d7

What version did you use? Version: v0.19.0

What config did you use? Config:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  opencensus:
  jaeger:
    protocols:
      grpc:
      thrift_http:
      thrift_compact:
      thrift_binary:
    remote_sampling:
      strategy_file: "/etc/jaeger/jaeger-sampling-config.json"
      # Doesn't work
      # fetch_endpoint: "jaeger-collector.observability:5778"
  zipkin:
  prometheus:
    config:
      - job_name: ambassador/ambassador/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - ambassador
        metrics_path: /metrics
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_service
          regex: ambassador-admin
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: ambassador-admin
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: ambassador-admin
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: auth/keycloak/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - auth
        scrape_interval: 30s
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: metrics
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: keycloak
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: keycloak
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-management
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http-management
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/grafana0/0
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        scrape_interval: 1m
        scrape_timeout: 30s
        metrics_path: /metrics
        scheme: http
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: grafana0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: grafana
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: service
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_grafana0
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: service
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-apiserver/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - default
        scheme: https
        tls_config:
          insecure_skip_verify: true
          server_name: kubernetes
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_component
          regex: apiserver
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_provider
          regex: kubernetes
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: https
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_component
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: https
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-coredns/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: prom0-kube-prometheus-coredns
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: http-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-kube-proxy/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: kube-proxy
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: http-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-kube-scheduler/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: prom0-kube-prometheus-kube-scheduler
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_component
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: http-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-kubelet/0
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        scheme: https
        tls_config:
          insecure_skip_verify: true
          server_name: kubernetes
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_k8s_app
          regex: kubelet
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: https-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: https-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
        metric_relabel_configs:
        - source_labels:
          - node
          target_label: instance
          action: replace
      - job_name: monitoring/prom0-kube-prometheus-kubelet/1
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - kube-system
        metrics_path: /metrics/cadvisor
        scheme: https
        tls_config:
          insecure_skip_verify: true
          server_name: kubernetes
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_k8s_app
          regex: kubelet
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: https-metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_k8s_app
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: https-metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-operator/0
        honor_labels: true
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: operator
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: prom0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: kube-prometheus
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-kube-prometheus-prometheus/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        metrics_path: /metrics
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_component
          regex: prometheus
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: prom0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: kube-prometheus
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: monitoring/prom0-node-exporter/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
          namespaces:
            names:
            - monitoring
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_instance
          regex: prom0
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_app_kubernetes_io_name
          regex: node-exporter
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - source_labels:
          - __meta_kubernetes_service_label_jobLabel
          target_label: job
          regex: (.+)
          replacement: ${1}
        - target_label: endpoint
          replacement: metrics
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: kubernetes_node
          regex: (.*)
          replacement: $1
          action: replace
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: observability/otel-collector-monitor/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_service
          regex: otel-collector
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: metrics
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: metrics
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
      - job_name: web/static-web-monitor/0
        honor_labels: false
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          source_labels:
          - __meta_kubernetes_service_label_service
          regex: static-web-frontend
        - action: keep
          source_labels:
          - __meta_kubernetes_endpoint_port_name
          regex: http
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Node;(.*)
          replacement: ${1}
          target_label: node
        - source_labels:
          - __meta_kubernetes_endpoint_address_target_kind
          - __meta_kubernetes_endpoint_address_target_name
          separator: ;
          regex: Pod;(.*)
          replacement: ${1}
          target_label: pod
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: service
        - source_labels:
          - __meta_kubernetes_pod_name
          target_label: pod
        - source_labels:
          - __meta_kubernetes_pod_container_name
          target_label: container
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: job
          replacement: ${1}
        - target_label: endpoint
          replacement: http
        - source_labels:
          - __address__
          target_label: __tmp_hash
          modulus: 1
          action: hashmod
        - source_labels:
          - __tmp_hash
          regex: 0
          action: keep
  # k8s_cluster:
  # auth_type: "serviceAccount"
  # collection_interval: 10s
  # node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable]
processors:
  batch:
  memory_limiter:
    # Same as --mem-ballast-size-mib CLI argument
    # ballast_size_mib: 128
    # 80% of maximum memory up to 2G
    limit_mib: 450
    # 25% of limit up to 2G
    spike_limit_mib: 128
    check_interval: 5s
  k8s_tagger:
    passthrough: true
extensions:
  health_check: {}
  # https://github.com/open-telemetry/opentelemetry-specification/blob/main/experimental/trace/zpages.md
  zpages: {}
  pprof: {}
exporters:
  otlp/lightstep:
    endpoint: "ingest.lightstep.com:443"
    headers: {"lightstep-access-token": ${lightstepAccessToken}}
  # jaeger_thrift:
  #   url: "http://jaeger-collector.observability:14268/api/traces"
  logging:
    loglevel: info
  kafka:
    protocol_version: 2.0.0
    brokers:
      - "kafka0-headless.kafka:9092"
service:
  extensions: [health_check, zpages, pprof]
  pipelines:
    metrics:
      receivers: [otlp, opencensus]
      processors: [memory_limiter, batch]
      exporters: [otlp/lightstep]
    traces/1:
      receivers: [otlp, zipkin, jaeger]
      processors: [memory_limiter, batch, k8s_tagger]
      exporters: [otlp/lightstep]

Environment OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): GKE

Additional context The crash started when I added the prometheus receiver, everything else worked fine before that.

yeya24 commented 3 years ago

This is a configuration problem. You need to add scrape_configs: in your prometheus config section.

  prometheus:
    config:
      scrape_configs:
      - job_name: ambassador/ambassador/0
        honor_labels: false
      ...
DanTulovsky commented 3 years ago

oh.. duh, sorry. But perhaps the binary shouldn't crash on an invalid config? Or does it do that on purpose?

bogdandrutu commented 3 years ago

It does that on purpose, we are working to add a dry-run flag just to validate config soon.