search

bluecmd / fortigate_exporter

Prometheus exporter for Fortigate firewalls
GNU General Public License v3.0
232 stars 72 forks source link

Fortigate-exporter exception error report #266

Closed happinesslijian closed 8 months ago

happinesslijian commented 9 months ago

I have filtered as follows, it was fine at first, but after waiting for a while, the following problem occurred. I'm not sure how to solve this problem. I read related issues. My prometheus fetches every 30 seconds.

global:
  scrape_interval: 30s
exclude:
  - System/SensorInfo
systemctl status fortigate-exporter.service 
● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 2h 8min ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 11.1M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de0e0}, 0xc000314100)
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc000711620?}, {0x9d8f98?, 0xc0000de0e0?}, 0x6?)
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c000, {0x9d9b08, 0xc0001b61b0})
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 03 12:37:57 prometheus-server fortigate-exporter[4159665]: 2024/01/03 12:37:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.583 seconds

This bothers me a lot, I don’t know what the problem is 😣Fortigate exporter service sometimes works well and sometimes bad

systemctl status fortigate-exporter.service 
● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 3h 0min ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 11.9M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 03 13:24:52 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:24:52 Probe of "https://10.0.0.1:10443" succeeded, took 2.689 seconds
1月 03 13:25:25 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:25:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.588 seconds
1月 03 13:25:50 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:25:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.633 seconds
1月 03 13:26:22 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:26:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.693 seconds
1月 03 13:26:50 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:26:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.641 seconds
1月 03 13:27:26 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:27:26 Probe of "https://10.0.0.1:10443" succeeded, took 7.109 seconds
1月 03 13:27:55 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:27:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.636 seconds
1月 03 13:28:25 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:28:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.628 seconds
1月 03 13:28:57 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:28:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.624 seconds
1月 03 13:29:25 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:29:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.615 seconds
systemctl status fortigate-exporter.service 
● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 3h 5min ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 9.1M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0001a1800)
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc00033ecc0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00023e3f0, {0x9d9b08, 0xc0001b61b0})
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb

The intuitive feeling is that the fortigate-exporter service is normal for a while and abnormal for a while. I'm not sure how to solve this problem. Need your help thanks.

bluecmd commented 9 months ago

Can you attach the full log? Right now it is only possible to see half the error stack trace.

happinesslijian commented 9 months ago

Can you attach the full log? Right now it is only possible to see half the error stack trace.

To reiterate, my version is as follows:

Fortigate_exporter Version: 1.23.2 Fortigate Version: v7.0.12 prometheus Version:v2.47.0

Now I use systemctl --no-pager status fortigate-exporter.service | lessto view the information of fortigate-exporter.service, as follows:

● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 23h ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 11.8M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0001fc200)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0003e6630?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c3f0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:07:51 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:07:51 Probe of "https://10.0.0.1:10443" succeeded, took 2.123 seconds
1月 04 10:08:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:08:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.603 seconds

Then I used the command journalctl -u fortigate-exporter.service -n 200 | less and got the following information:

-- Logs begin at Sat 2023-10-21 12:56:43 CST, end at Thu 2024-01-04 10:15:55 CST. --
1月 04 10:03:22 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:03:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.332 seconds
1月 04 10:03:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:03:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.626 seconds
1月 04 10:04:26 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:04:26 Probe of "https://10.0.0.1:10443" succeeded, took 7.122 seconds
1月 04 10:04:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:04:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.612 seconds
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:20 http: panic serving 192.168.16.5:59748: runtime error: slice bounds out of range [1:0]
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: goroutine 115345 [running]:
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc00046c2e8?})
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0005a4000}, 0xc0000a04b0?)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0001f0300, {0x9d9bb0, 0xc00059e000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc00059c000)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc00059c000)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0001b50e0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc0001ac3f0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:05:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.642 seconds
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:50 http: panic serving 192.168.16.5:53386: runtime error: slice bounds out of range [1:0]
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: goroutine 115999 [running]:
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc00046c438?})
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc00060a000}, 0xc00046c798?)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0006040f0, {0x9d9bb0, 0xc000608000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc000600000)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc000600000)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004c4300?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc0001ac480, {0x9d9b08, 0xc0001b61b0})
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:05:57 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.406 seconds
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:20 http: panic serving 192.168.16.5:52678: runtime error: slice bounds out of range [1:0]
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: goroutine 116489 [running]:
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0000a0408?})
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc000106000}, 0xc0000a1950?)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0001f0318, {0x9d9bb0, 0xc000652310}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc000684000)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc000684000)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0005240f0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c2d0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:20 Probe of "https://10.0.0.1:10443" succeeded, took 0.615 seconds
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:50 http: panic serving 192.168.16.5:56724: runtime error: slice bounds out of range [1:0]
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: goroutine 116493 [running]:
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc00046c018?})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0003e4000}, 0xc00046c618?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00027e2d0, {0x9d9bb0, 0xc0003e2000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc000684400)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc000684400)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004ae030?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c360, {0x9d9b08, 0xc0001b61b0})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:50 http: panic serving 192.168.16.5:49372: runtime error: slice bounds out of range [1:0]
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: goroutine 116606 [running]:
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0001e0048?})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0003e40c0}, 0xc00046c5b8?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc000602138, {0x9d9bb0, 0xc000106000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de0e0}, 0xc000684000)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de0e0?}, 0x6a79ba?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de0e0}, 0xc000684000)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004d44e0?}, {0x9d8f98?, 0xc0000de0e0?}, 0x6?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc0001ac7e0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:07:20 http: panic serving 192.168.16.5:60994: runtime error: slice bounds out of range [1:0]
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: goroutine 116689 [running]:
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0001e0120?})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0003e4000}, 0xc0000267c8?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00040e6a8, {0x9d9bb0, 0xc0000b0460}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc0001fc200)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0001fc200)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0003e6630?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c3f0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:07:51 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:07:51 Probe of "https://10.0.0.1:10443" succeeded, took 2.123 seconds
1月 04 10:08:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:08:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.603 seconds
1月 04 10:08:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:08:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.593 seconds
1月 04 10:09:22 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:09:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.986 seconds
1月 04 10:09:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:09:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.633 seconds
1月 04 10:10:27 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:10:27 Probe of "https://10.0.0.1:10443" succeeded, took 7.369 seconds
1月 04 10:10:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:10:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.573 seconds
1月 04 10:11:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:11:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.650 seconds
1月 04 10:11:57 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:11:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.466 seconds
1月 04 10:12:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:12:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.583 seconds
1月 04 10:12:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:12:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.611 seconds
1月 04 10:13:22 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:13:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.736 seconds
1月 04 10:13:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:13:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.597 seconds
1月 04 10:14:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:14:20 Probe of "https://10.0.0.1:10443" succeeded, took 0.655 seconds
1月 04 10:14:57 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:14:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.956 seconds
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:20 http: panic serving 192.168.16.5:44628: runtime error: slice bounds out of range [1:0]
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: goroutine 116688 [running]:
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0000261c8?})
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc00013e000}, 0xc0001e0018?)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc000010240, {0x9d9bb0, 0xc00064ff10}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004475c0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c6c0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:20 Probe of "https://10.0.0.1:10443" succeeded, took 0.593 seconds
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:55 http: panic serving 192.168.16.5:39620: runtime error: slice bounds out of range [1:0]
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: goroutine 117308 [running]:
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0000261c8?})
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc00067e000}, 0xc0001e0090?)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc000784108, {0x9d9bb0, 0xc000659b90}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc00058b740?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00023e120, {0x9d9b08, 0xc0001b61b0})
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:55 Probe of "https://10.0.0.1:10443" succeeded, took 0.591 seconds

By the way, besides the above two ways to view logs, are there any other ways? @bluecmd

bluecmd commented 9 months ago

Perfect, thanks!

Here is the issue at hand: runtime error: slice bounds out of range [1:0]. This is happening at github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage, line system_resources_usage.go:63.

https://github.com/bluecmd/fortigate_exporter/blob/99e11c024ae91b9880a3f893621469b70d5d4bd2/pkg/probe/system_resources_usage.go#L61-L66

Seems like sr.Results.CPU might contain no specific CPU values occasionally in your case.

@happinesslijian Are you comfortable patching and compiling the fortigate exporter yourself? I would suggest adding a print before this code so we can see how the API response looks like when things break for you. Alternatively, you can try to fetch https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global when the issue happens and see if you can catch the abnormal API response that way.

happinesslijian commented 9 months ago

很好，谢谢！

这是当前的问题：runtime error: slice bounds out of range [1:0]。 这发生在github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsagesystem_resources_usage.go:63 行。

https://github.com/bluecmd/fortigate_exporter/blob/99e11c024ae91b9880a3f893621469b70d5d4bd2/pkg/probe/system_resources_usage.go#L61-L66

在您的情况下，似乎sr.Results.CPU有时可能不包含特定的 CPU 值。

@happinesslijian您是否愿意自己修补和编译 fortigate 导出器？我建议在此代码之前添加打印内容，以便我们可以看到当您出现问题时 API 响应的样子。 或者，您可以尝试https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global在问题发生时进行获取，看看是否可以通过这种方式捕获异常的 API 响应。

@bluecmd I've done it and compiled it as you asked, adding the print operation on line 61
The Fortigate-exporter file has been compiled and has been run in my environment. The latest log is as follows:

● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-01-05 12:45:37 CST; 2min 46s ago
   Main PID: 300608 (fortigate-expor)
      Tasks: 10 (limit: 19119)
     Memory: 10.0M
     CGroup: /system.slice/fortigate-exporter.service
             └─300608 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0003e7050, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 sr.Results.CPU: [{0} {0} {0} {0} {0} {0} {2} {0} {0}]
1月 05 12:47:53 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:53 Probe of "https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global" failed, took 2.589 seconds
1月 05 12:48:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 sr.Results.CPU: [{3} {2} {2} {2} {2} {6} {5} {4} {11}]
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 Probe of "https://10.0.0.1:10443" failed, took 0.478 seconds
-- Logs begin at Sat 2023-10-21 12:56:43 CST, end at Fri 2024-01-05 12:49:02 CST. --
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc00036e780?}, {0x9c8df0, 0xc000182460}, 0xc000171900)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0004760a0, {0x9c9458, 0xc000128b40})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:20 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:20 http: panic serving 192.168.16.5:55700: runtime error: slice bounds out of range [1:0]
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: goroutine 5276 [running]:
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0000287e0})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc00010e0c0}, 0xc0000284b0?)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0003ecac8, {0x9c9420, 0xc000531800}, {0xc000028438?, 0xc0000833a8?}, 0xc0002a0384?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182380}, 0xc00040cb00)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc000149af0?, {0x9c8df0?, 0xc000182380?}, 0x0?)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc0002a038b?, {0x9c8df0, 0xc000182380}, 0xc00040cb00)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc0000c5080?}, {0x9c8df0, 0xc000182380}, 0xc00040cb00)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0000b8f00, {0x9c9458, 0xc000128b40})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:13:49 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:13:50 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:50 Probe of "https://10.0.0.1:10443" failed, took 0.433 seconds
1月 05 12:14:19 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:20 http: panic serving 192.168.16.5:43206: runtime error: slice bounds out of range [1:0]
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: goroutine 5294 [running]:
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0005321e0})
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc000744000}, 0xc000028180?)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0004a41e0, {0x9c9420, 0xc000190ba0}, {0xc0000d1848?, 0xc0000833a8?}, 0xc000524004?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182380}, 0xc00078b800)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc00041baf0?, {0x9c8df0?, 0xc000182380?}, 0x0?)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc00052400b?, {0x9c8df0, 0xc000182380}, 0xc00078b800)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc00018f5f0?}, {0x9c8df0, 0xc000182380}, 0xc00078b800)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc00016a960, {0x9c9458, 0xc000128b40})
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:20 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:21 http: panic serving 192.168.16.5:52160: runtime error: slice bounds out of range [1:0]
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: goroutine 5325 [running]:
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0000d0780})
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc0007440c0}, 0xc0000d0798?)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0004667b0, {0x9c9420, 0xc000359e60}, {0xc000028270?, 0xc0000833a8?}, 0xc000454044?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182460}, 0xc000000200)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc000145af0?, {0x9c8df0?, 0xc000182460?}, 0x0?)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc00045404b?, {0x9c8df0, 0xc000182460}, 0xc000000200)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc00064e3f0?}, {0x9c8df0, 0xc000182460}, 0xc000000200)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0000b8b40, {0x9c9458, 0xc000128b40})
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:14:49 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:50 http: panic serving 192.168.16.5:52766: runtime error: slice bounds out of range [1:0]
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: goroutine 5373 [running]:
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0000d0600})
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc00010e0c0}, 0xc0000d0858?)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0000ac6a8, {0x9c9420, 0xc00007d260}, {0xc0000d07c8?, 0xc0000833a8?}, 0xc000524284?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182380}, 0xc000170700)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc00014baf0?, {0x9c8df0?, 0xc000182380?}, 0x0?)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc00052428b?, {0x9c8df0, 0xc000182380}, 0xc000170700)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc000212720?}, {0x9c8df0, 0xc000182380}, 0xc000170700)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0000b8dc0, {0x9c9458, 0xc000128b40})
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:14:50 prometheus-server systemd[1]: Stopping fortigate-exporter...
1月 05 12:14:50 prometheus-server systemd[1]: fortigate-exporter.service: Succeeded.
1月 05 12:14:50 prometheus-server systemd[1]: Stopped fortigate-exporter.
1月 05 12:14:50 prometheus-server systemd[1]: Started fortigate-exporter.
1月 05 12:14:50 prometheus-server systemd[1]: fortigate-exporter.service: Main process exited, code=exited, status=203/EXEC
1月 05 12:14:50 prometheus-server systemd[1]: fortigate-exporter.service: Failed with result 'exit-code'.
1月 05 12:45:37 prometheus-server systemd[1]: Started fortigate-exporter.
1月 05 12:45:37 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:37 FortigateExporter (devel) ( (no hash) )
1月 05 12:45:37 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:37 Loaded 1 API keys
1月 05 12:45:37 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:37 Fortigate exporter running, listening on ":9710"
1月 05 12:45:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:50 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:45:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:50 sr.Results.CPU: [{0} {0} {1} {1} {0} {0} {1} {1} {0}]
1月 05 12:45:53 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:53 Probe of "https://10.0.0.1:10443" failed, took 3.314 seconds
1月 05 12:46:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:46:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:20 sr.Results.CPU: [{2} {0} {3} {1} {1} {3} {1} {4} {3}]
1月 05 12:46:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:20 Probe of "https://10.0.0.1:10443" failed, took 0.454 seconds
1月 05 12:46:49 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:46:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:50 sr.Results.CPU: [{1} {1} {0} {1} {0} {1} {1} {1} {0}]
1月 05 12:46:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:50 Probe of "https://10.0.0.1:10443" failed, took 0.444 seconds
1月 05 12:47:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 sr.Results.CPU: []
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 http: panic serving 192.168.16.5:59590: runtime error: slice bounds out of range [1:0]
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: goroutine 5 [running]:
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve.func1()
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1854 +0xbf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: panic({0x9008c0, 0xc0005843a8})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/runtime/panic.go:890 +0x263
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9ea7a0, 0xc0002b8000}, 0xc000028348?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/system_resources_usage.go:64 +0x7a5
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0002b60f0, {0x9ee298, 0xc00031c120}, 0xc00023a160?, 0xc00032b728?, {0xc000186540, {0x9317d2, 0x5}, 0x1e, 0xa, ...})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/probe.go:179 +0x9cf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9edec0, 0xc000106000}, 0xc0002f2400)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.HandlerFunc.ServeHTTP(0xc0001b0080?, {0x9edec0?, 0xc000106000?}, 0x40da8a?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2122 +0x2f
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*ServeMux).ServeHTTP(0xc00011c14b?, {0x9edec0, 0xc000106000}, 0xc0002f2400)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2500 +0x149
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.serverHandler.ServeHTTP({0xc0001a2270?}, {0x9edec0, 0xc000106000}, 0xc0002f2400)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2936 +0x316
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0001cc000, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 sr.Results.CPU: []
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 http: panic serving 192.168.16.5:51508: runtime error: slice bounds out of range [1:0]
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: goroutine 189 [running]:
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve.func1()
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1854 +0xbf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: panic({0x9008c0, 0xc000584288})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/runtime/panic.go:890 +0x263
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9ea7a0, 0xc0002b8180}, 0xc0000281f8?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/system_resources_usage.go:64 +0x7a5
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0005a8360, {0x9ee298, 0xc00016ebd0}, 0xc000151e40?, 0xc0001d6728?,
 {0xc000186540, {0x9317d2, 0x5}, 0x1e, 0xa, ...})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/probe.go:179 +0x9cf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9edec0, 0xc000106460}, 0xc0001aef00)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.HandlerFunc.ServeHTTP(0xc0001b0480?, {0x9edec0?, 0xc000106460?}, 0x40da8a?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2122 +0x2f
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*ServeMux).ServeHTTP(0xc0004004cb?, {0x9edec0, 0xc000106460}, 0xc0001aef00)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2500 +0x149
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.serverHandler.ServeHTTP({0xc00016ea20?}, {0x9edec0, 0xc000106460}, 0xc0001aef00)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2936 +0x316
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0003e67e0, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:49 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 sr.Results.CPU: []
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 http: panic serving 192.168.16.5:53810: runtime error: slice bounds out of range [1:0]
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: goroutine 223 [running]:
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve.func1()
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1854 +0xbf
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: panic({0x9008c0, 0xc000028360})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/runtime/panic.go:890 +0x263
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9ea7a0, 0xc0003bc000}, 0xc000028390?)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/system_resources_usage.go:64 +0x7a5
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0005a8438, {0x9ee298, 0xc0003fb4a0}, 0xc000304420?, 0xc00078b728?, {0xc000186540, {0x9317d2, 0x5}, 0x1e, 0xa, ...})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/probe.go:179 +0x9cf
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9edec0, 0xc0001069a0}, 0xc00036ea00)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.HandlerFunc.ServeHTTP(0xc00026a280?, {0x9edec0?, 0xc0001069a0?}, 0x40da8a?)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2122 +0x2f
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*ServeMux).ServeHTTP(0xc00040060b?, {0x9edec0, 0xc0001069a0}, 0xc00036ea00)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2500 +0x149
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.serverHandler.ServeHTTP({0xc0003fb3b0?}, {0x9edec0, 0xc0001069a0}, 0xc00036ea00)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2936 +0x316
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0003e7050, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 sr.Results.CPU: [{0} {0} {0} {0} {0} {0} {2} {0} {0}]
1月 05 12:47:53 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:53 Probe of "https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global" failed, took 2.589 seconds
1月 05 12:48:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 sr.Results.CPU: [{3} {2} {2} {2} {2} {6} {5} {4} {11}]
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 Probe of "https://10.0.0.1:10443" failed, took 0.478 seconds
1月 05 12:48:49 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:48:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:50 sr.Results.CPU: [{0} {0} {1} {1} {1} {0} {0} {1} {0}]
1月 05 12:48:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:50 Probe of "https://10.0.0.1:10443" failed, took 0.436 seconds

I try to access https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global in the browser and get the following results:

{
   "http_method":"GET",
   "results":{
   },
   "vdom":"root",
   "path":"system",
   "name":"resource",
   "action":"usage",
   "status":"success",
   "serial":"FG200FT922925682",
   "version":"v7.0.13",
   "build":566
}
{
  "http_method":"GET",
  "results":{
    "cpu":[
      {
        "current":0,
        "historical":{
          "1-min":{
            "values":[
              [
                1704430083000,
                0
              ],
              [
                1704430080000,
                0
              ],
              [
                1704430077000,
                1
              ],
···
···
···
              [
                1704350510000,
                0
              ],
              [
                1704346190000,
                0
              ]
            ],
            "max":0,
            "min":0,
            "average":0,
            "start":1704346190000,
            "end":1704430084000
          }
        }
      }
    ]
  },
  "vdom":"root",
  "path":"system",
  "name":"resource",
  "action":"usage",
  "status":"success",
  "serial":"FG200FT922925682",
  "version":"v7.0.13",
  "build":566
}

Please help me check where the problem occurs. Thank you. @bluecmd

bluecmd commented 9 months ago

From your investigations, this seems to be the weird response:

{
   "http_method":"GET",
   "results":{
   },
   "vdom":"root",
   "path":"system",
   "name":"resource",
   "action":"usage",
   "status":"success",
   "serial":"FG200FT922925682",
   "version":"v7.0.13",
   "build":566
}

This seems like a bug in FortiOS producing empty resources. We can ignore those replies, or we can retry the fetch.

Can you try the patch in PR #269 to see if it makes things work any better?

happinesslijian commented 9 months ago

From your investigations, this seems to be the weird response:

{
   "http_method":"GET",
   "results":{
   },
   "vdom":"root",
   "path":"system",
   "name":"resource",
   "action":"usage",
   "status":"success",
   "serial":"FG200FT922925682",
   "version":"v7.0.13",
   "build":566
}

This seems like a bug in FortiOS producing empty resources. We can ignore those replies, or we can retry the fetch.

Can you try the patch in PR #269 to see if it makes things work any better?

Hello, @bluecmd I have rebuilt Fortigate-exporter according to your method and it is running. The problem still exists. The detailed log is as follows:

● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-01-05 22:27:55 CST; 21min ago
   Main PID: 369115 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 9.8M
     CGroup: /system.slice/fortigate-exporter.service
             └─369115 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de24b?, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002de090?}, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca000, {0x9bf318, 0xc00046a150})
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:49:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:49:22 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:22 Probe of "https://10.0.0.1:10443" failed, took 2.404 seconds
-- Logs begin at Sat 2023-10-21 12:56:43 CST, end at Fri 2024-01-05 22:50:18 CST. --
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:43:50 Got system resources, will retry fetch..
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:43:50 http: panic serving 192.168.16.5:39056: runtime error: index out of range [0] with length 0
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: goroutine 1289 [running]:
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0001c60a8})
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0002da0c0}, 0xc0000d4378?)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000c510, {0x9bf2e0, 0xc0004729c0}, 0x0?, 0x2?, {0xc>
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000238500)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0002101cb?, {0x9beea8, 0xc0004dc380}, 0xc000238500)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc00039e420?}, {0x9beea8, 0xc0004dc380}, 0xc000238500)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc00035a0a0, {0x9bf318, 0xc00046a150})
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:44:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:44:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:20 Got system resources, will retry fetch..
1月 05 22:44:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:20 Got system resources, will retry fetch..
1月 05 22:44:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:20 Got system resources, will retry fetch..
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:21 Error: Out of attempts to fetch system resources, giving up
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:21 http: panic serving 192.168.16.5:59406: runtime error: index out of range [0] with length 0
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: goroutine 1310 [running]:
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0001c6108})
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001fc000}, 0xc0005a1690?)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000dc98, {0x9bf2e0, 0xc0004e73e0}, 0x0?, 0xb?, {0xc>
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004600e0}, 0xc000238800)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004600e0?}, 0x40d425?)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0002102cb?, {0x9beea8, 0xc0004600e0}, 0xc000238800)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002b6930?}, {0x9beea8, 0xc0004600e0}, 0xc000238800)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0000bee60, {0x9bf318, 0xc00046a150})
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:44:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:44:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:50 Got system resources, will retry fetch..
1月 05 22:44:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:50 Got system resources, will retry fetch..
1月 05 22:44:52 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:52 Probe of "https://10.0.0.1:10443" failed, took 3.111 seconds
1月 05 22:45:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:45:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:20 Probe of "https://10.0.0.1:10443" failed, took 0.418 seconds
1月 05 22:45:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:45:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:50 Probe of "https://10.0.0.1:10443" failed, took 0.359 seconds
1月 05 22:46:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:46:22 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:22 Probe of "https://10.0.0.1:10443" failed, took 2.352 seconds
1月 05 22:46:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:46:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:50 Got system resources, will retry fetch..
1月 05 22:46:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:50 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Error: Out of attempts to fetch system resources, giving up
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 http: panic serving 192.168.16.5:42864: runtime error: index out of range [0] with length 0
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: goroutine 1361 [running]:
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0000d4420})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001e8000}, 0xc000024210?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0001d00f0, {0x9bf2e0, 0xc0000b7ec0}, 0x0?, 0xc0006136>
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004601c0}, 0xc000127800)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004601c0?}, 0x40d425?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de14b?, {0x9beea8, 0xc0004601c0}, 0xc000127800)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002de120?}, {0x9beea8, 0xc0004601c0}, 0xc000127800)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca5a0, {0x9bf318, 0xc00046a150})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Error: Out of attempts to fetch system resources, giving up
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 http: panic serving 192.168.16.5:46512: runtime error: index out of range [0] with length 0
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: goroutine 1530 [running]:
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0000240d8})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001e80c0}, 0xc0005989e0?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000cea0, {0x9bf2e0, 0xc0006c37a0}, 0x0?, 0xb?, {0xc>
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000338a00)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de60b?, {0x9beea8, 0xc0004dc380}, 0xc000338a00)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0004a8150?}, {0x9beea8, 0xc0004dc380}, 0xc000338a00)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0000bf0e0, {0x9bf318, 0xc00046a150})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:47:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Got system resources, will retry fetch..
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Got system resources, will retry fetch..
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Got system resources, will retry fetch..
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Error: Out of attempts to fetch system resources, giving up
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 http: panic serving 192.168.16.5:40386: runtime error: index out of range [0] with length 0
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: goroutine 1540 [running]:
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0004d6240})
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001d4000}, 0xc0004d6408?)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00058c930, {0x9bf2e0, 0xc0004e6cc0}, 0x0?, 0xb?, {0xc>
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc460}, 0xc000126400)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc460?}, 0x40d425?)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de2cb?, {0x9beea8, 0xc0004dc460}, 0xc000126400)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc000167b30?}, {0x9beea8, 0xc0004dc460}, 0xc000126400)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca0a0, {0x9bf318, 0xc00046a150})
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:47:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Got system resources, will retry fetch..
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Got system resources, will retry fetch..
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Got system resources, will retry fetch..
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Error: Out of attempts to fetch system resources, giving up
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 http: panic serving 192.168.16.5:49440: runtime error: index out of range [0] with length 0
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: goroutine 1601 [running]:
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc000024168})
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc00032a000}, 0xc0004d6300?)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000c618, {0x9bf2e0, 0xc0006c26c0}, 0x0?, 0xb?, {0xc>
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000310800)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de70b?, {0x9beea8, 0xc0004dc380}, 0xc000310800)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc000385410?}, {0x9beea8, 0xc0004dc380}, 0xc000310800)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0000befa0, {0x9bf318, 0xc00046a150})
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:48:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:20 Got system resources, will retry fetch..
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:20 http: panic serving 192.168.16.5:48902: runtime error: index out of range [0] with length 0
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: goroutine 1593 [running]:
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc000024228})
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc00038c000}, 0xc000024228?)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00058c678, {0x9bf2e0, 0xc000498900}, 0x0?, 0xb?, {0xc>
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc460}, 0xc000126b00)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc460?}, 0x40d425?)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc00027e44b?, {0x9beea8, 0xc0004dc460}, 0xc000126b00)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0001b4cf0?}, {0x9beea8, 0xc0004dc460}, 0xc000126b00)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc000122780, {0x9bf318, 0xc00046a150})
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:48:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:50 Got system resources, will retry fetch..
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:50 http: panic serving 192.168.16.5:45376: runtime error: index out of range [0] with length 0
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: goroutine 1629 [running]:
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc000024150})
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc00038c0c0}, 0xc000024138?)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0002121b0, {0x9bf2e0, 0xc0004e6600}, 0x0?, 0xc0004646>
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de24b?, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002de090?}, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca000, {0x9bf318, 0xc00046a150})
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:49:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:49:22 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:22 Probe of "https://10.0.0.1:10443" failed, took 2.404 seconds
1月 05 22:49:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:49:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:50 Probe of "https://10.0.0.1:10443" failed, took 0.414 seconds
happinesslijian commented 8 months ago

Hello, @bluecmd is there any other solution? My problem here still exists.😣

bluecmd commented 8 months ago

@happinesslijian Given the API responses you have shown there is nothing the exporter can do - your fortigate returns invalid data. As long as it does that, we cannot produce metrics.

I would suggest opening an issue with Fortinet and/or upgrading FortiOS version and see if the problem goes away.