scraping metrics with a dockerized prometheus

[stefangweichinger]

systemd service

Downloaded current binary into:

ls -l /usr/local/sbin/dnsbl-exporter 
-rwxr-xr-x 1 root docker 10285056 May  6  2023 /usr/local/sbin/dnsbl-exporter

Set up service file as in https://github.com/Luzilla/dnsbl_exporter/issues/86#issue-860439522

modified paths to ini-files etc ->

# cat /etc/systemd/system/dnsbl-exporter.service 
[Unit]
Description=DNSBL Exporter
StartLimitBurst=5

[Service]
User=root
#ExecStart=/usr/local/sbin/dnsbl-exporter --config.dns-resolver 127.0.0.1 --config.rbls /etc/prometheus/dnsbl-exporter/rbls.ini --config.targets /etc/prometheus/dnsbl-exporter/targets.ini
ExecStart=/usr/local/sbin/dnsbl-exporter --config.rbls /etc/prometheus/dnsbl-exporter/rbls.ini --config.targets /etc/prometheus/dnsbl-exporter/targets.ini
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=default.target

#  ls -l /etc/prometheus/dnsbl-exporter/
total 8
-rw-r--r-- 1 root root 2764 Nov 12 09:22 rbls.ini
-rw-r--r-- 1 root root  182 Nov 12 09:24 targets.ini

Service runs, and returns metrics:

 curl http://localhost:9211/metrics
# HELP luzilla_rbls_duration The scrape's duration (in seconds)
# TYPE luzilla_rbls_duration gauge
luzilla_rbls_duration 0.002717003
# HELP luzilla_rbls_ips_blacklisted Blacklisted IPs
# TYPE luzilla_rbls_ips_blacklisted gauge

[..]

luzilla_rbls_ips_blacklisted{hostname="oc.oops.co.at",ip="45.84.138.128",rbl="ix.dnsbl.manitu.net"} 0

[..]

# HELP luzilla_rbls_listed The number of listings in RBLs (this is bad)
# TYPE luzilla_rbls_listed gauge
luzilla_rbls_listed{rbl="ix.dnsbl.manitu.net"} 0
luzilla_rbls_listed{rbl="pbl.spamhaus.org"} 0
luzilla_rbls_listed{rbl="sbl.spamhaus.org"} 0
luzilla_rbls_listed{rbl="xbl.spamhaus.org"} 0
luzilla_rbls_listed{rbl="zen.spamhaus.org"} 0
# HELP luzilla_rbls_targets The number of targets that are being probed (configured via targets.ini or ?target=)
# TYPE luzilla_rbls_targets gauge
luzilla_rbls_targets 4
# HELP luzilla_rbls_used The number of RBLs to check IPs against (configured via rbls.ini)
# TYPE luzilla_rbls_used gauge
luzilla_rbls_used 5
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0

prometheus

Prometheus runs in docker, so I have to enable it to access ports on the host level. This is done by setting "extra_hosts":

services:

  prometheus:
    image: prom/prometheus:v2.47.2
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    networks:
      - back-tier
      - front-tier
    restart: always
    extra_hosts:
      - "host.docker.internal:host-gateway"

The scraping is configured in prometheus.yml:

  - job_name: 'dnsbl-exporter'

    static_configs:
      - targets: ['host.docker.internal:9211']

No errors are shown, but I don't see any metrics with "luzilla*" in Grafana/Prometheus.

[stefangweichinger]

Checking things further:

docker exec -it prometheus-prometheus-1 sh
telnet host.docker.internal 9211
telnet: can't connect to remote host (172.17.0.1): Connection refused

Interesting ...

I use the same hostname for mysqld-exporter, that one works:

telnet host.docker.internal 9104
Connected to host.docker.internal

So it might be related to dnsbl_exporter and maybe its service-file.

[till]

I forgot, is there a telnet -v to see what it does?

Otherwise, maybe: nc -zv host.docker.internal 9211? Also, not entirely sure if host.docker.internal in your setup includes the loop back interface?

Can you curl http://0.0.0.0:9211/metrics from the host? To make sure it listens on all expected interfaces.

[stefangweichinger]

There is no telnet -v, sorry.

And nc returns nothing:

/prometheus $ nc -zv host.docker.internal 9211
/prometheus $

The curl works from the host, yes:

 curl http://0.0.0.0:9211/metrics
# HELP luzilla_rbls_duration The scrape's duration (in seconds)
# TYPE luzilla_rbls_duration gauge
luzilla_rbls_duration 0.147485518
# HELP luzilla_rbls_ips_blacklisted Blacklisted IPs
# TYPE luzilla_rbls_ips_blacklisted gauge
luzilla_rbls_ips_blacklisted{hostname="imap.my.tld",ip="185.xxxx",rbl="ix.dnsbl.manitu.net"} 0

[stefangweichinger]

I tested with binding dnsbl_exporter to "0.0.0.0:9211" or "127.0.0.1:9211" on the host, no difference.

Checked the docker networks, FQDN host.docker.internal is reachable from the prometheus container as seen above (the mysqld_exporter can be scraped from host AND prometheus container).

Interesting issue somehow ...

[till]

You verified the binding? E.g. lsof -i? I can't think of a reason why it wouldn't work.

Can you share which Linux, Docker, etc. you run? This seems like an interesting problem.

[stefangweichinger]

Yes, I verfied the binding with ss -lnp | grep 9211 (not the most competent use, I assume ;-) )

lsof -i | grep 9211
dnsbl-exp 3203757       root    3u  IPv6 28037996      0t0  TCP *:9211 (LISTEN)

It's not that urgent on my side and I am quite busy this week. I'll get back to it asap.

This is running on docker 20.10.24+dfsg1 (Debian 12.1). Maybe it's some strange iptables-issue and magically solved by a reboot.

[stefangweichinger]

I have a second mailserver where I will also try to set up your exporter etc. Then we will see if it works there (and maybe, why).

[stefangweichinger]

Solved: iptables rule was missing! What a shame ... took me quite a long time. Now I get metrics into prometheus and can proceed. Thanks, closing here.