Unable to scrape Postgres cert metrics

atmanohar commented 3 years ago

Host operating system: output of `uname -a`

blackbox_exporter version: output of `blackbox_exporter --version`

What is the blackbox.yml module config.

modules:
  tls_connect:
    prober: tcp
    timeout: 60s
    tcp:
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      query_response:
        - expect: "OK.*STARTTLS"
        - send: ". STARTTLS"
        - expect: "OK"
        - starttls: true
        - send: ". capability"
        - expect: "CAPABILITY IMAP4rev1"
      tls: true
      tls_config:
        insecure_skip_verify: true

  ldap:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: "ip4"
      query_response:
        - send: !!binary "MCICAQFgHQIBAwQIbm90YXVzZXKADm5vdHRoZXBhc3N3b3Jk"
        - expect: !!binary "MAwCAQFhBw=="

  postgres:
    prober: tcp
    tcp:
      query_response:
        - send: !!binary AAAACATSFi8=
        - starttls: true

What is the prometheus.yml scrape config.

- job_name: postgres-blackbox
    honor_timestamps: true
    scrape_interval: 5m
    scrape_timeout: 5m
    metrics_path: /probe
    params:
      module: [tcp_connect] ##also tried with modules ldap and postgres

    kubernetes_sd_configs:
    - api_server: null
      role: service
      namespaces:
        names: []
    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace]
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      target_label: kubernetes_name
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: prometheus-blackbox-exporter.do-prom-athira.svc.cluster.local:9115

What logging output did you get from adding `&debug=true` to the probe URL?

What did you do that produced an error?

What did you expect to see?

# HELP probe_ssl_earliest_cert_expiry Returns earliest SSL cert expiry date
# TYPE probe_ssl_earliest_cert_expiry gauge
probe_ssl_earliest_cert_expiry 1.655444464e+09
# HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in unixtime
# TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge
probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
# HELP probe_ssl_last_chain_info Contains SSL leaf certificate information
# TYPE probe_ssl_last_chain_info gauge

What did you see instead?

Logs for the probe:
ts=2021-06-18T11:49:24.361029282Z caller=main.go:304 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Beginning probe" probe=tcp timeout_seconds=60
ts=2021-06-18T11:49:24.361147413Z caller=tcp.go:41 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Resolving target address" ip_protocol=ip4
ts=2021-06-18T11:49:24.364193802Z caller=tcp.go:41 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Resolved target address" ip=240.224.66.208
ts=2021-06-18T11:49:24.364219851Z caller=tcp.go:122 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=info msg="Dialing TCP with TLS"
ts=2021-06-18T11:49:24.366345309Z caller=main.go:119 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=error msg="Error dialing TCP" err="tls: first record does not look like a TLS handshake"
ts=2021-06-18T11:49:24.366385816Z caller=main.go:304 module=tls_connect target=blackblock-hippo-svc.do-blackbox-postgres.svc:8009 level=error msg="Probe failed" duration_seconds=0.00527054

tobiasbp commented 3 years ago

I'm also wondering if blackbox_exporter supports checking of certs used by PostgreSQL?

openssl supports the postgreqsl protocol nowadays: https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html#testing-protocols-that-upgrade-to-ssl

ful09003 commented 3 years ago

👋 just adding my experience with this problem. At my workplace, we have need to monitor an internal SSL/TLS-enabled Postgres. After a lot of unsuccessful attempts to coerce blackbox_exporter into doing that for us (as we use it everywhere else), we decided to instead add functionality into an in-house SSL/TLS management daemon to monitor this for us (with Postgres in mind). It would be great to some day remove that functionality and leverage blackbox_exporter for this need, too!

For the blackbox_exporter maintainers or others looking to do this with an established Golang ecosystem: our functionality for monitoring Postgres began with investigating how this worked, and improving it for our platform: https://github.com/chr4/pg-check-cert. I hope it can serve as a good starting point for how to do this solely in Go (or maybe there are more recent/idiomatic examples somewhere out there 🤷)

tarvip commented 3 years ago

Alternatively you can now use ssl_exporter to scrape postgres cert metrics. Postgres support is available since version v2.3.1.

tacho commented 1 year ago

I also encountered this, and the issue is that currently the blackbox exporter query response expect matches operate in line mode, whereas PostgreSQL sends a single byte as a response to the SSLRequest, so it never matches and the exporter times out. The linked PR fixes this and adds a working example. Using it, I have been able to successfully scrape Postgres certificate metrics.

prometheus / blackbox_exporter