search

bluecmd / spectrum_virtualize_exporter

Prometheus exporter for IBM Spectrum Virtualize compatible SANs
8 stars 8 forks source link

No outgoing packets to Spektrum target and thus getting no metrics #10

Closed DirkFries closed 2 years ago

DirkFries commented 2 years ago

Hi @all, I built the exporter and basically it does run, but I can not see any outgoing packets to the system that should be monitored.

My yaml File looks like this:

"http://v3700:8443":
  user: username
  password: password

If I start the exporter like this ...

[prometheus@prometheus-t prometheus]# /home/prometheus/spectrum_virtualize_exporter -auth-file /home/prometheus/monitor.yaml -scrape-timeout 30 -insecure
2022/02/25 15:20:47 Loaded 1 API credentials
2022/02/25 15:20:47 Spectrum Virtualize exporter running, listening on ":9747"

... it is able to read the monitor.yaml file and read my API configuration, but after this it just hangs.

If I do a tcpdump to see any packets between my machine and the Spektrum targets I see not a single packet. What does work is the webserver. If I connect to my machine port 9747 I get

 go_gc_duration_seconds{quantile="0"} 3.0896e-05
 go_gc_duration_seconds{quantile="0.25"} 3.0896e-05
 go_gc_duration_seconds{quantile="0.5"} 0.000165137
 go_gc_duration_seconds{quantile="0.75"} 0.006196516
 go_gc_duration_seconds{quantile="1"} 0.006196516
 go_gc_duration_seconds_sum 0.006392549
 go_gc_duration_seconds_count 3
 go_goroutines 8
 go_info{version="go1.13.8"} 1
 go_memstats_alloc_bytes 2.26024e+06
 go_memstats_alloc_bytes_total 6.39332e+06
 go_memstats_buck_hash_sys_bytes 1.443582e+06
 go_memstats_frees_total 98602
 go_memstats_gc_cpu_fraction 2.554158460560801e-05
 go_memstats_gc_sys_bytes 2.377728e+06
 go_memstats_heap_alloc_bytes 2.26024e+06
 go_memstats_heap_idle_bytes 6.2767104e+07
 go_memstats_heap_inuse_bytes 3.948544e+06
 go_memstats_heap_objects 10113
 go_memstats_heap_released_bytes 6.2767104e+07
 go_memstats_heap_sys_bytes 6.6715648e+07
 go_memstats_last_gc_time_seconds 1.645778475426287e+09
 go_memstats_lookups_total 0
 go_memstats_mallocs_total 108715
 go_memstats_mcache_inuse_bytes 3472
 go_memstats_mcache_sys_bytes 16384
 go_memstats_mspan_inuse_bytes 50864
 go_memstats_mspan_sys_bytes 81920
 go_memstats_next_gc_bytes 4.194304e+06
 go_memstats_other_sys_bytes 733690
 go_memstats_stack_inuse_bytes 393216
 go_memstats_stack_sys_bytes 393216
 go_memstats_sys_bytes 7.1762168e+07
 go_threads 7
 process_cpu_seconds_total 0.05
 process_max_fds 1024
 process_open_fds 7
 process_resident_memory_bytes 1.5015936e+07
 process_start_time_seconds 1.64577814476e+09
 process_virtual_memory_bytes 5.00002816e+08
 process_virtual_memory_max_bytes 1.8446744073709552e+19
 promhttp_metric_handler_requests_in_flight 1
 promhttp_metric_handler_requests_total{code="200"} 2
 promhttp_metric_handler_requests_total{code="500"} 0
 promhttp_metric_handler_requests_total{code="503"} 0

I have also prepared an strace file that might be helpful:

file.txt

Tested both on Ubuntu 22.04 and Almalinux 8 with the same behavior. Expected behavior: Seeing outgoing to the spektrum target and, if configured properly, seeing metrics of that target at the output of the webserver.

Does anyone have a hint what is wrong ?

Bye, Dirk

bluecmd commented 2 years ago

Hello.

The exporter Is a black box style exporter, you need to configure it as a target on a Prometheus instance for it to start polling. If you Google around a bit there should be plenty of material how they work. Happy to accept a PR with an example when you have one

DirkFries commented 2 years ago

Hello,

thank you very much for your answer. Of course the next step would be to scrape http://myhost:9747/metrics with my prometheus instance to get the metrics into prometheus. But I think it makes no difference if http://myhost:9747/metrics is opened by a prometheus or a browser. My problem is, that the spektrum_virtualize_exporter does even't try to connect to the targets if I open up http://myhost:9747/metrics

Bye, Dirk

bluecmd commented 2 years ago

The endpoint /metrics is for exporter-internal metrics. You need to call /probe. Try http://myhost:9747/probe?target=http://v3700:8443.

Like this:

- job_name: spectrum_virtualize
  metrics_path: /probe
  scheme: http
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (?:.+)(?::\/\/)([^:]*).*
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: 'your-spectrum-exporter:9747'
    action: replace
  static_configs:
  - targets:
    - https://your-v7000-here:7443

As I mentioned before, this is a black box exporter - not a regular exporter. On black box exporters /metrics is not really useful unless you are debugging the exporter itself.

DirkFries commented 2 years ago

Ashes on my head ! Now I know that mean with "black box".

Thanks a lot for your hint ! :-)