search

canonical / blackbox-exporter-k8s-operator

This charmed operator automates the operational procedures of running Blackbox Exporter, a Prometheus exporter running blackbox probes over a multitude of protocols.
https://charmhub.io/blackbox-exporter-k8s
Apache License 2.0
0 stars 3 forks source link

scrape_config's `module` parameter does not seem to be passed to the prometheus #11

Closed mkalcok closed 6 months ago

mkalcok commented 11 months ago

Bug Description

module parameter does not seem to be properly passed to the Prometheus when scrape job is created.

To Reproduce

  1. \<deploy cos>
  2. juju deploy --channel=edge blackbox-exporter-k8s
  3. juju config blackbox-exporter-k8s config_file='@./config.yaml'
  4. juju relate blackbox-exporter-k8s prometheus
  5. juju config blackbox-exporter-k8s probes_file='@./probes.yaml'

config.yaml:

modules:
  https_2xx:
    http:
      fail_if_not_ssl: true
      follow_redirects: true
      method: GET
      tls_config:
        insecure_skip_verify: false
      valid_http_versions:
      - HTTP/1.1
      - HTTP/2.0
      valid_status_codes: []
    prober: http
    timeout: 5s

probes.yaml:

scrape_configs:
  - job_name: "blackbox"
    metrics_path: /probe
    params:
      module: [https_2xx]
    static_configs:
      - targets:
        - https://example.org

Environment

juju snap: 3.2.3 juju controller: 3.2.2 blackbox-exporter-k8s char: rev. 2 prometheus char: rev. 103

Relevant log output

Scrape job configuration as seen in Prometheus:

- job_name: juju_cos-external-targets_8a913a02_blackbox-exporter-k8s_prometheus_scrape_blackbox
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  follow_redirects: true
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: probe_target
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: blackbox-exporter-k8s-0.blackbox-exporter-k8s-endpoints.cos-external-targets.svc.cluster.local:9115
    action: replace
  - source_labels: [juju_model, juju_model_uuid, juju_application]
    separator: _
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  static_configs:
  - targets:
    - https://example.org
    labels:
      juju_application: blackbox-exporter-k8s
      juju_charm: blackbox-exporter-k8s
      juju_model: cos-external-targets
      juju_model_uuid: 8a913a02-10e0-4acf-8582-fe1d07e9faf9

Additional context

As can be seen in the Prometheus' job configuration, module parameter was not configured. Correct configuration should contain

  params:
    module:
    - https_2xx
nobuto-m commented 9 months ago

I can confirm this. There is no info about the custom module name in the relation data.

  - relation-id: 43
    endpoint: metrics-endpoint
    related-endpoint: self-metrics-endpoint
    application-data:
      alert_rules: '{"groups": [{"name": "cos_1e96fd62_blackbox-exporter_blackbox-exporter-self-monitoring_alerts",
        "rules": [{"alert": "BlackboxExporterUnitIsUnavailable", "expr": "up{} < 1",
        "for": "0m", "labels": {"severity": "critical", "juju_model": "cos", "juju_model_uuid":
        "1e96fd62-8236-46ef-899e-c73471bd6705", "juju_application": "blackbox-exporter",
        "juju_charm": "blackbox-exporter-k8s"}, "annotations": {"summary": "Blackbox
        Exporter unit {{ $labels.juju_model }}/{{ $labels.juju_unit }} unavailable",
        "description": "The Blackbox Exporter unit {{ $labels.juju_model }} {{ $labels.juju_unit
        }} is unavailable LABELS = {{ $labels }}\n"}}, {"alert": "BlackboxExporterUnitIsDown",
        "expr": "avg_over_time(up{}[1m]) < 0.5", "for": "0m", "labels": {"severity":
        "critical", "juju_model": "cos", "juju_model_uuid": "1e96fd62-8236-46ef-899e-c73471bd6705",
        "juju_application": "blackbox-exporter", "juju_charm": "blackbox-exporter-k8s"},
        "annotations": {"summary": "Blackbox Exporter unit {{ $labels.juju_model }}/{{
        $labels.juju_unit }} has been unreachable at least 50% of the time over the
        last minute", "description": "The Blackbox Exporter unit {{ $labels.juju_model
        }} {{ $labels.juju_unit }} has been unreachable at least 50% of the time over
        the last minute LABELS = {{ $labels }}\n"}}]}, {"name": "cos_1e96fd62_blackbox-exporter_blackbox_missing_alerts",
        "rules": [{"alert": "BlackboxJobMissing", "expr": "absent(up{})", "for": "0m",
        "labels": {"severity": "warning", "juju_model": "cos", "juju_model_uuid":
        "1e96fd62-8236-46ef-899e-c73471bd6705", "juju_application": "blackbox-exporter",
        "juju_charm": "blackbox-exporter-k8s"}, "annotations": {"summary": "Blackbox
        job missing (instance {{ $labels.instance }})", "description": "A Blackbox
        job has disappeared\nVALUE = {{ $value }}\nLABELS = {{ $labels }}\n"}}]}]}'
      scrape_jobs: '[{"metrics_path": "/metrics", "static_configs": [{"targets": ["blackbox-exporter-0.blackbox-exporter-endpoints.cos.svc.cluster.local:9115"]}]},
        {"metrics_path": "/probe", "static_configs": [{"targets": ["http://ubuntu.com",
        "https://ubuntu.com", "https://releases.ubuntu.com", "https://non-existent.canonical.com/",
        "https://192.168.151.105:16443"]}], "job_name": "blackbox_k8s_endpoints",
        "relabel_configs": [{"source_labels": ["__address__"], "target_label": "__param_target"},
        {"source_labels": ["__param_target"], "target_label": "instance"}, {"source_labels":
        ["__param_target"], "target_label": "probe_target"}, {"target_label": "__address__",
        "replacement": "blackbox-exporter-0.blackbox-exporter-endpoints.cos.svc.cluster.local:9115"}]}]'
      scrape_metadata: '{"model": "cos", "model_uuid": "1e96fd62-8236-46ef-899e-c73471bd6705",
        "application": "blackbox-exporter", "unit": "blackbox-exporter/0", "charm_name":
        "blackbox-exporter-k8s"}'
    related-units:
      blackbox-exporter/0:
        in-scope: true
        data:
          egress-subnets: 10.152.183.122/32
          ingress-address: 10.152.183.122
          private-address: 10.152.183.122
          prometheus_scrape_unit_address: 10.1.63.27
          prometheus_scrape_unit_name: blackbox-exporter/0
  provider-id: prometheus-0
  address: 10.1.63.24
lucabello commented 6 months ago

Cannot reproduce with the latest charm revision; the scrape job now correctly contains a new module https_2xx_new I used to try to reproduce the issue:

params:
    module:
    - https_2xx_new

Closing!