probe_success 0 after setting trusted hosts for admin users (not rest api admin)

Dear all,

I just ran into issues not really knowing where to continue or what I'm doing wrong. The exporter itself is running and the probe also worked until I rebooted / upgraded my monitoring server from debian 10 to debian 11.

Changing some hostnames and URLs - but I always tried with IP and hostname.

When I try to curl the exporter locally, the probe ends after roughly 30 seconds - same output for when I try via firefox on my pc with http://mon01:9710/probe?target=https://fortigate01.domain.de:

root@mon01:# curl 'localhost:9710/probe?target=https://fortigate01.domain.de
# HELP probe_duration_seconds How many seconds the probe took to complete
# TYPE probe_duration_seconds gauge
probe_duration_seconds 30.000529618
# HELP probe_success Whether or not the probe succeeded
# TYPE probe_success gauge
probe_success 0

This request can NOT be seen in the fortigates API logs.

Server mon01 can ping the fortigate:

root@mon01:# ping fortigate01.domain.de
PING fortigate01.domain.de (10.0.0.254) 56(84) bytes of data.
64 bytes from fortigate01.domain.de (10.0.0.254): icmp_seq=1 ttl=255 time=0.242 ms

When I try to reach the fortigate API from my laptops browser, thats not an issue, So I know the API user itself should be working:

visiting site: https:/fortigate01.domain.de/api/v2/monitor/system/status/?access_token=xxxxxxxxxxxxxx
shows:
{
  "http_method":"GET",
  "results":{
    "model_name":"FortiGate",
    "model_number":"100F",
    "model":"FG100F",
    "hostname":"removed",
    "log_disk_status":"not_available"
  },
  "vdom":"root",
  "path":"system",
  "name":"status",
  "action":"",
  "status":"success",
  "serial":"removed",
  "version":"removed",
  "build":removed
}

This request can also be seen on the fortigate in the API logs from API user "monitoring".

In some other issue I found on how to go from here, so I tried to also curl this page from my server mon01:

root@mon01:# curl -v -X GET -I -k https://fortigate01.domain.de/api/v2/monitor/system/status/?access_token=xxxxxxxxxxxxxx
*   Trying 10.0.0.254:443...
* connect to 10.0.0.254 port 443 failed: Connection timed out
* Failed to connect to fortigate01.domain.de port 443: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to fortigate01.domain.de port 443: Connection timed out

But from there it seems to be broken.

My fortigate-key.yaml looks like this:

"https://fortigate01.domain.de":
  token: xxxxxxxxxxxxxx
  probes:
    include:
      - System
      - VPN
      - Firewall/Policies
      - VirtualWAN
      - WebUI
      - User
    exclude:
      - System/SensorInfo

My service file for fortigate_exporter like this - already tried adding higher timeout values, but also without them no chance:

[Unit]
Description=Fortigate Exporter

[Service]
User=prometheus
ExecStart=/path/to/fortigate_exporter -auth-file /path/to/fortigate-key.yaml -scrape-timeout 120 -https-timeout 120
Restart=always

[Install]
WantedBy=multi-user.target

Exporter is running, as I said, but giving me errors each scrape - always after 10 seconds:

● fortigate_exporter.service - Fortigate Exporter
     Loaded: loaded (/path/to/fortigate_exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-02-14 18:27:12 CET; 43min ago
   Main PID: 1046 (fortigate_expor)
      Tasks: 5 (limit: 9492)
     Memory: 8.1M
        CPU: 167ms
     CGroup: /system.slice/fortigate_exporter.service
             └─1046 /path/to/fortigate_exporter -auth-file /path/to/fortigate-key.yaml -scrape-timeout 120 -https-timeout 120

Feb 14 19:10:19 monitoring fortigate_exporter[1046]: 2024/02/14 19:10:19 Error: API connectivity test failed, Get "https://fortigate01.domain.de/api/v2/monitor/system/status": context canceled
Feb 14 19:10:19 monitoring fortigate_exporter[1046]: 2024/02/14 19:10:19 Probe of "https://fortigate01.domain.de" failed, took 10.001 seconds

What does context canceled mean here?

bluecmd / fortigate_exporter

probe_success 0 after setting trusted hosts for admin users (not rest api admin) #281