prometheus scraping data but nothing is recieved.

sebasdt commented 1 year ago

Hi, At the end of this comment there is a temporally fix that worked for me

I just stumbled into this awesome project.

My setup: Ive got a single node setup with proxmox and 3 lxc containers for testing. All of the containers are installed using proxmox helper scripts container 1: is the prometheus container 2: is the wireguard with exporter container 3: grafana for dashboards. Everything is set to default and no ports are changed or firewalls enabled. The wireguard vpn has 2 clients both of em can connect and use the internet.

What works: Prometheus is able to talk to the wireguard exporter and is trying to collect data. I used this command to get the container to start: sudo docker run -d -p 9586:9586 -it mindflavor/prometheus-wireguard-exporter -v true The issue: Im running into the following issue where prometheus is scraping data from the exporter but nothing is recieved back. While looking at the container logs with docker logs --since=1h pensive_jepsen returns this:

[2023-02-15T08:46:49Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::render_with_names(self == WireGuard { interfaces: {} }, pehm == None, options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false } called
[2023-02-15T08:46:51Z TRACE prometheus_exporter_base] serve_function:: req.uri() == /metrics, req.method() == GET
[2023-02-15T08:46:51Z TRACE prometheus_exporter_base] received headers ==>
    host => 127.0.0.1:9586
    user-agent => curl/7.68.0
    accept => */*
[2023-02-15T08:46:51Z TRACE prometheus_exporter_base] serve_function:: options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false }
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] interfaces_to_handle == ["all"]
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] peer_entry_hashmap == None
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] wg show all dump stdout ==
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] wg show all dump stderr ==
[2023-02-15T08:46:51Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::try_from() called
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter::wireguard] WireGuard { interfaces: {} }
[2023-02-15T08:46:51Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::render_with_names(self == WireGuard { interfaces: {} }, pehm == None, options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false } called
[2023-02-15T08:46:51Z TRACE prometheus_exporter_base] serve_function:: req.uri() == /metrics, req.method() == GET
[2023-02-15T08:46:51Z TRACE prometheus_exporter_base] received headers ==>
    host => 127.0.0.1:9586
    user-agent => curl/7.68.0
    accept => */*
[2023-02-15T08:46:51Z TRACE prometheus_exporter_base] serve_function:: options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false }
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] interfaces_to_handle == ["all"]
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] peer_entry_hashmap == None
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] wg show all dump stdout ==
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter] wg show all dump stderr ==
[2023-02-15T08:46:51Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::try_from() called
[2023-02-15T08:46:51Z TRACE prometheus_wireguard_exporter::wireguard] WireGuard { interfaces: {} }
[2023-02-15T08:46:51Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::render_with_names(self == WireGuard { interfaces: {} }, pehm == None, options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false } called
[2023-02-15T08:46:58Z TRACE prometheus_exporter_base] serve_function:: req.uri() == /metrics, req.method() == GET
[2023-02-15T08:46:58Z TRACE prometheus_exporter_base] received headers ==>
    host => wire.home.lan:9586
    user-agent => Prometheus/2.42.0
    accept => application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1
    accept-encoding => gzip
    x-prometheus-scrape-timeout-seconds => 10
[2023-02-15T08:46:58Z TRACE prometheus_exporter_base] serve_function:: options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false }
[2023-02-15T08:46:58Z TRACE prometheus_wireguard_exporter] interfaces_to_handle == ["all"]
[2023-02-15T08:46:58Z TRACE prometheus_wireguard_exporter] peer_entry_hashmap == None
[2023-02-15T08:46:58Z TRACE prometheus_wireguard_exporter] wg show all dump stdout ==
[2023-02-15T08:46:58Z TRACE prometheus_wireguard_exporter] wg show all dump stderr ==
[2023-02-15T08:46:58Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::try_from() called
[2023-02-15T08:46:58Z TRACE prometheus_wireguard_exporter::wireguard] WireGuard { interfaces: {} }
[2023-02-15T08:46:58Z DEBUG prometheus_wireguard_exporter::wireguard] WireGuard::render_with_names(self == WireGuard { interfaces: {} }, pehm == None, options == Options { verbose: true, prepend_sudo: false, separate_allowed_ips: false, extract_names_config_files: None, interfaces: None, export_remote_ip_and_port: false, export_latest_handshake_delay: false } called

Directly dumping the wireguard data with sudo wg show all dump shows it can export data.

wg0     REDACTED=   REDACTEDU=    57172   off
wg0     REDACTED=  REDACTED=    188.207.75.211:7681                                                                                                          10.66.66.2/32,fd42:42:42::2/128  1676451176      302012  1965500 off
wg0     REDACTED=    REDACTED=    (none)  10.66.66.3/32,fd42:42:42::3/128                                                                                      00       0       off

Then my next step was to test if exporter collects anything with curl -s http://127.0.0.1:9586/metrics on the host machine.

# HELP wireguard_sent_bytes_total Bytes sent to the peer
# TYPE wireguard_sent_bytes_total counter

# HELP wireguard_received_bytes_total Bytes received from the peer
# TYPE wireguard_received_bytes_total counter

# HELP wireguard_latest_handshake_seconds UNIX timestamp seconds of the last handshake
# TYPE wireguard_latest_handshake_seconds gauge

Possible cause of the Issue: It all seems like the exporter isnt able to collect data so I came up with these theories:

The docker exporter container isn't able to collect data from wireguard on the host machine.
its using the wrong or no wireguard interface.

If you need more information about anything related feel free to ask. Im all new to prometheus and want to learnt the basics.

Temporary fix: Add -a true -v true at the end of the docker command after the docker image. Now it should work and scrape data. The command should look like something like this: sudo docker run -d --net=host --cap-add=NET_ADMIN --restart unless-stopped --name wgexporter mindflavor/prometheus-wireguard-exporter -v true -a true

jano477 commented 1 year ago

+1 I'm having the same issue. I used 3.6.1 because of: https://github.com/MindFlavor/prometheus_wireguard_exporter/issues/99

sebasdt commented 1 year ago

Okay @jano477 I have found a work-around and that is this:

In the docks at the bottom there they say you need to add -a true -v true at the end of the docker command after the image. Like this: sudo docker run -d --net=host --cap-add=NET_ADMIN --name wgexporter mindflavor/prometheus-wireguard-exporter -v true -a true

doing this should make it work atleast for me. Its kinda fixed for now but not out of the box!

jano477 commented 1 year ago

Okay, I see. It works when I run it via docker run like docker run -d --volume /etc/wireguard/wg0.conf:/etc/wireguard/wg0.conf --net=host --cap-add=NET_ADMIN --name wg-exporter mindflavor/prometheus-wireguard-exporter -n /etc/wireguard/wg0.conf -i wg0 -v true -a true

I couldn't get it working via compose-file and still don't know why.

  wg-exporter:
    image: mindflavor/prometheus-wireguard-exporter
    container_name: wg-exporter
    volumes:
      - /etc/wireguard/:/etc/wireguard/:ro
    environment:
      - PROMETHEUS_WIREGUARD_EXPORTER_PORT=9587
      - PROMETHEUS_WIREGUARD_EXPORTER_INTERFACES=wg0
      - PROMETHEUS_WIREGUARD_EXPORTER_EXPORT_REMOTE_IP_AND_PORT_ENABLED=true
      - PROMETHEUS_WIREGUARD_EXPORTER_CONFIG_FILE_NAMES=/etc/wireguard/wg0.conf
      - EXPORT_LATEST_HANDSHAKE_DELAY=true
    network_mode: host
    cap_add:
      - NET_ADMIN
    command: -v true -a true

That one results not exporting any metrics. When I exec into the container I can see the wg-command is not working due to permission problems on wg0. It doesn't change when the container runs privileged.

docker exec -it wg-exporter /bin/sh
/usr/local/bin $ wg
Unable to access interface wg0: Operation not permitted

docker logs wg-exporter
[2023-02-25T18:39:20Z INFO  prometheus_wireguard_exporter] prometheus_wireguard_exporter v3.6.6 starting...
[2023-02-25T18:39:20Z INFO  prometheus_wireguard_exporter] using options: Options { verbose: true, prepend_sudo: true, separate_allowed_ips: false, extract_names_config_files: Some(["/etc/wireguard/wg0.conf"]), interfaces: Some(["wg0"]), export_remote_ip_and_port: true, export_latest_handshake_delay: true }
[2023-02-25T18:39:20Z INFO  prometheus_wireguard_exporter] starting exporter on http://0.0.0.0:9587/metrics
[2023-02-25T18:39:20Z INFO  prometheus_exporter_base] Listening on http://0.0.0.0:9587/metrics

But after several minutes the container suddenly starts to export metrics despite the permission problem persists.

sebasdt commented 1 year ago

Also dont forget --restart unless-stopped variable somehow the container stops randomly without an logs.

sebasdt commented 1 year ago

Temporary fix: Add -a true -v true at the end of the docker command after the docker image. Now it should work and scrape data. The command should look like something like this: sudo docker run -d --net=host --cap-add=NET_ADMIN --restart unless-stopped --name wgexporter mindflavor/prometheus-wireguard-exporter -v true -a true

This has worked like a charm for me and fixed the issue. Closing the issue.

MindFlavor / prometheus_wireguard_exporter

prometheus scraping data but nothing is recieved. #109