Dynamically add hosts

[freedomwarrior]

Is it possible to add host from file like hosts.yml or something? Or only way is command args?

[SuperQ]

Currently on the command line. I have some ideas to add a yaml config with additional options per target. But it's only an idea at this point.

[xuanyuanaosheng]

@SuperQ @freedomwarrior May be this is a good solution: https://github.com/google/cloudprober

[cleonello]

+1 This really is a requirement for me to use smokeping_prober in a production environment.

[SuperQ]

@cleonello care to take a shot at adding the feature?

[cleonello]

@SuperQ, Sorry didn't see your reply. I unfortunately don't know go. I found a reasonable work around (for my deployments) for this issue, however. I specify an environment configuration file where I define the hosts to scrape which is then read in by my systemd smokeping_prober service. I do have to restart the service when the config is updated but it is suitable for my purposes.

For me to implement in production I still need to address issue #35. I need to add a custom label to my metrics in addition to IP address. We identify monitored equipment by unique ID's, not by IP address.

[freedomwarrior]

@SuperQ, Sorry didn't see your reply. I unfortunately don't know go. I found a reasonable work around (for my deployments) for this issue, however. I specify an environment configuration file where I define the hosts to scrape which is then read in by my systemd smokeping_prober service. I do have to restart the service when the config is updated but it is suitable for my purposes.

For me to implement in production I still need to address issue #35. I need to add a custom label to my metrics in addition to IP address. We identify monitored equipment by unique ID's, not by IP address.

Hi! That's how you do it?

EnvironmentFile=/etc/.progconf
ExecStart = /usr/bin/prog $ARG1 $ARG2

[edgarasg]

Would be great to have Consul support.

[paulfantom]

@SuperQ what about adding hosts/config in a similar way as it is done in blackbox-exporter? This way the existing Prober mechanism from prometheus-operator could be re-used to include smokeping_prober as another type of prober.

[SuperQ]

@paulfantom My current idea is something like this:

targets:
- host: foo.example.com
  network: ip4
  protocol: icmp
  labels:
    name: Loadbalancer VIP1 DC1 v4
- host: foo.example.com
  network: ip6
  protocol: udp
  labels:
    name: Loadbalancer VIP1 DC1 v6
- host: 10.0.0.1
  network: ip # Automatic ip4/ip6
  # protocol: icmp # Defaults to ICMP

Or maybe something like this:

targets:
- hosts:
  - foo.example.com
  - bar.example.com
   network: ip4
  protocol: icmp
  labels:
    name: Loadbalancer VIP1 DC1 v4
- hosts:
  - foo.example.com
  network: ip6
  protocol: udp
  labels:
    name: Loadbalancer VIP1 DC1 v6
- hosts:
 - 10.0.0.1
  network: ip # Automatic ip4/ip6
  # protocol: icmp # Defaults to ICMP

The blackbox-exporter takes the discovery from the Prometheus side via target param. This exporter has its own discovery.

[SuperQ]

Not dynamically adding, but the first step to allow config from a file.

https://github.com/SuperQ/smokeping_prober/pull/54

Once that is done and available, I'll re-work the pinger goroutine management in order to start/stop pingers on the fly.

[SuperQ]

One additional thought, on a reload/SIGHUP. I think the easiest / sanest behavior is that all pingers are stopped, and new ones are started.

This would reset all counters to zero, even those that existed before. Please comment if you think this is ok/not ok.

[baryluk]

Dynamic hosts via a file (similar to file_sd in prometheus) would be great, that is:

- targets:
    - foo
    - bar
  labels:
     xyz: fooish
  smokeping_options:
     protocol: icmp
     ...

Additional benefit of this format is that, for example it might be common to use blackbox_exporter or snmp_exporter and use file service-discovery. So essentially reusing essentially same format for smokeping prober would be nice, so tools that generate this file need minimal or no changes.

Of course the tricky part is to persist the counter values for the existing targets, if the content of the file changes, but effectively the target has same options and target. Having breaks in counters, on file reload, would be not good, as many people might be updating these files continuously every few minutes.

[SuperQ]

Of course the tricky part is to persist the counter values for the existing targets,

It's highly unlikely I will implement any kind of persistence. The most likely case is that the config reloader will be smart enough to not reset running probers that aren't changed.

[baryluk]

Of course the tricky part is to persist the counter values for the existing targets,

It's highly unlikely I will implement any kind of persistence. The most likely case is that the config reloader will be smart enough to not reset running probers that aren't changed.

Yes, this is exactly what I meant. Not persist on disk, just not restart probers that did not change.

I will take a look into code on weekend, and see if I can implement a prototype for this.

[jpds]

Could we also have this integrate with some form of service-discovery?

There's an example of blackbox-exporter here: https://github.com/cfunkhouser/tailscalesd?tab=readme-ov-file#example-pinging-tailscale-hosts