Adding InfluxDB support to smokeping

[mad-ady]

Hello all!

We're using smokeping to do some automated network parameter measurements to check on the perceived service quality we offer our end customers. We're using remote probes/slaves to do measurements from different points in the network and aggregating the data on the master. I've also created some plugins in the past few years, so we want to stick with smokeping for the measurements.

However, the web GUI is not really suitable for large scale data analysis. It's ok for historical analysis only. So, I'm planning on making a few (hopefully small) changes to smokeping to add support to write the measurements in an influxdb database, in addition to storing them as RRD. I hope to be able to push the changes (once they're working) to the main Smokeping branch. Having data in influxdb allows easier automated analysis (e.g. using an sql-like language to get the latest measurements for a destination from all the slaves, without having to parse the RRD files). It also opens up the possibility to have views exposed in Grafana and other more modern data visualization tools.

Here's what I'm planning:

1. Allow the addition of custom configuration options to the config file so that those options can be mapped to influxdb tags (so that they are easily searchable in influxdb). For instance, something like this:

   
   + Hosts_in_region_A
   menu = Hosts in region A
   title = Hosts in region A

++ client1 host = 192.168.1.1 probe = fping influx_download = 100Mbps influx_upload = 10Mbps influx_service_type = FTTB

For now, when I try to add configuration options that are not defined, I
get the error:

ERROR: /opt/smokeping/bin/../etc/config, line 144: unknown variable 'influx_download'

I'm guessing I need to bypass or extend the parser so that such errors
are ignored, or become warnings. Or, better yet, if I stick with the
prefix influx_ I can make it accept any influx_ parameter, but complain
on other typos (which is useful). I'll need to do some reading on the
Grammar module...

2. Add a new config section that defines connectivity to influxdb (URL,
user, password)

3. Add a hook inside update_rrds() to intercept the data before being
written to the RRD (before or after the call to RRDs::update). Push the
raw data to influxdb (either inline for efficiency, or with a new
function call for readability).

I'm not 100% certain that the update_rrds() from Smokeping.pm gets
called for slave updates as well, or there is an equivalent in the CGI
interface. I'm guessing it's the same, but I will do some tests when I
get there.

So - what are your thoughts on this? Are there better/easier ways of
doing it?

Thanks!

[mad-ady]

Ok, I was able to solve the second point - adding database connectivity options to the Smokeping config: https://github.com/mad-ady/SmokePing/commit/cced5ce20a9c18821c04a0104a7c19d0d79bd38d

But I'm lost (reading the code) about how I can allow optional parameters that match a regex for each probe configuration... I'm guessing I need to look somewhere in get_parser: https://github.com/oetiker/SmokePing/blob/master/lib/Smokeping.pm#L2116

    # The _dyn() stuff here is quite confusing, so here's a walkthrough:
    # 1   Probe is defined in the Probes section
    # 1.1 _dyn is called for the section to add the probe- and target-specific
    #     vars into the grammar for this section and its subsections (subprobes)
    # 1.2 A _dyn sub is installed for all mandatory target-specific variables so 
    #     that they are made non-mandatory in the Targets section if they are
    #     specified here. The %storedtargetvars hash holds this information.
    # 1.3 If a probe section has any subsections (subprobes) defined, the main
    #     section turns into a template that just offers default values for
    #     the subprobes. Because of this a _dyn sub is installed for subprobe
    #     sections that makes any mandatory variables in the main section non-mandatory.
    # 1.4 A similar _dyn sub as in 1.2 is installed for the subprobe target-specific
    #     variables as well.
    # 2   Probe is selected in the Targets section top
    # 2.1 _dyn is called for the section to add the probe- and target-specific
    #     vars into the grammar for this section and its subsections. Any _default
    #     values for the vars are removed, as they will be propagated from the Probes
    #     section.
    # 2.2 Another _dyn sub is installed for the 'probe' variable in target subsections
    #     that behaves as 2.1
    # 2.3 A _dyn sub is installed for the 'host' variable that makes the mandatory
    #     variables mandatory only in those sections that have a 'host' setting.
    # 2.4 A _sub sub is installed for the 'probe' variable in target subsections that
    #     bombs out if 'probe' is defined after any variables that depend on the
    #     current 'probe' setting.

I agree, this is confusing. How do I wrap my head around it?

In the mean time I will try to start pushing data from Smokeping to influx (actually use the DB)

[oetiker]

look at https://metacpan.org/pod/Config::Grammar to see how this config system works

[Strykar]

This sounds sweet, I'm curious if you have considered adding Alias or Label support for hostnames used in Smokeping's config? Prometheus uses this (honor_labels) to have one job with multiple targets under it, in its YAML config:

- job_name: 'netdata-scrape'
    metrics_path: '/api/v1/allmetrics'
    params:
      format: [prometheus]
    honor_labels: true
    static_configs:
    - targets: ['i-5.host.com:19999']
    - targets: ['localhost:19999']

For example, this is useful when ingesting similar data from segregated nodes feeding Netdata to Prometheus.

[mad-ady]

My plan is to add support for custom configuration items (that start with influx_) that will be added into the database as keys. That way you can generate the smokeping config with all the relevant information you need in influx to do filtering, etc.

On Thu, Oct 24, 2019 at 8:18 PM Avinash H. Duduskar < notifications@github.com> wrote:

This sounds sweet, I'm curious if you have considered adding Alias or Label support for hostnames used in Smokeping's config? Prometheus uses this (honor_labels) to have one job with multiple targets under it, in its YAML config:

• job_name: 'netdata-scrape' metrics_path: '/api/v1/allmetrics' params: format: [prometheus] honor_labels: true static_configs:
  • targets: ['i-5.host.com:19999']
  • targets: ['localhost:19999']

For example, this is useful when ingesting similar data from segregated nodes feeding Netdata https://github.com/netdata/netdata to Prometheus.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=ABAO6KXOUUTZZQOZHECKJ3DQQHKG7A5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECFY75Q#issuecomment-546017270, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KTOQXPJHI2IAMYQKN3QQHKG7ANCNFSM4I6AUBOA .

[mad-ady]

Time for some progress report. TL;DR: it works, but needs polishing.

Managed to implement pushing data to influxdb. Also I docker-ized a smokeping master/slave install based on my branch. Data gets written to influxdb fine, but I need to expose more keys so that filtering would be easier. I'm still lost when it comes to Config::Grammar - @oetiker, maybe you can give me a tip in the right direction (exposing optional influx_* key/value pairs in any configuration section). I plan on sending a PR after merging these changes as well

How to test current changes: build smokeping from my branch here: https://github.com/mad-ady/SmokePing, or get the docker images from here (it will download the correct image for your architecture): https://hub.docker.com/repository/docker/madady/smokeping-influx. On this page I have some test docker-compose scripts to set up a master and a slave instance: https://github.com/mad-ady/docker-smokeping I need to work more on documentation...

Now, to enable influx export, you'll need to add the following section to your smokeping config:

*** InfluxDB ***

host = 192.168.100.100
database = smokeping
timeout = 10
port = 8086

You need to create a database like so (assumes influx is running in a container, like everything else does this decade):

root@iq-management:~# docker exec -it influxdb bash  
root@5d53c9e39e75:/# influx
Connected to http://localhost:8086 version 1.7.6
InfluxDB shell version: 1.7.6
Enter an InfluxQL query
> create database smokeping;
> exit
root@5d53c9e39e75:/# exit

To check that data is inserted in influxdb, you can run queries like:

select * from FPing

I also set up a minimal Grafana dashboard that uses influx data. It's generic enough to get you started, but needs polishing (like suppressing lines where data is missing, but connecting adjacent dots). Here is the template: https://pastebin.com/qrHNXExZ

Here is how it looks like:

I'm showing only min/max/average and loss percent, though all measurements (ping*) are stored in influx for further processing.

Suggestions and testing are more than welcome!

[oetiker]

since you already have the code, I guess best is to prepare a PR then we can discuss details based on that.

[mad-ady]

Ok, I'll send one tomorrow. What about adding documentation/examples? I filled in the pod descriptions, but not much else.

On Thu, Dec 19, 2019, 17:17 Tobias Oetiker notifications@github.com wrote:

since you already have the code, I guess best is to prepare a PR then we can discuss details based on that.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=ABAO6KUMKFKPPHY3LDXRHDDQZOGCPA5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJ5TWY#issuecomment-567531995, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KS7VNGIQOUG2CUEHMDQZOGCPANCNFSM4I6AUBOA .

[oetiker]

well documentation is in the pod ... so ... maybe it is all that is needed ...

[Strykar]

@mad-ady Nice work! Some dashboard inspiration in case you haven't seen these already:

1. https://grafana.com/grafana/dashboards/11335
2. https://gist.github.com/RichiH/5cc979fc910af0294579e0ee040317ea
3. https://hveem.no/visualizing-latency-variance-with-grafana
4. https://peter.run/blog/2019-07-28-visualising-latency-variance-in-grafana-in-2019/ (Different take)

[mad-ady]

Thanks! Third link looks better. I used the fourth one as a basis for my template.

I haven't submitted the patches yet - I want to add "slave" information, and I'll also be looking at adding custom variables to influxdb so that filtering is made easier. I also want to troubleshoot some data that isn't pushed to influx yet (speedtests data and youtube-dl data). Not sure what's wrong with it...

On Thu, Dec 26, 2019 at 1:23 AM Avinash H. Duduskar < notifications@github.com> wrote:

@mad-ady https://github.com/mad-ady Nice work! Some dashboard inspiration in case you haven't seen these already:

1. https://grafana.com/grafana/dashboards/11335
2. https://gist.github.com/RichiH/5cc979fc910af0294579e0ee040317ea
3. https://hveem.no/visualizing-latency-variance-with-grafana

4. https://peter.run/blog/2019-07-28-visualising-latency-variance-in-grafana-in-2019/ (Different take)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=ABAO6KUI7QHXVMAXHPJ22QTQ2PTOVA5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHUUY6Y#issuecomment-568937595, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KXBJHMLANRZAU7TOZTQ2PTOVANCNFSM4I6AUBOA .

[sgroom-lb]

Looking forward to this... I've used smokeping for ever but this would allow me to do some remote polling and push it back to my tig stack @ our hq location.

Hope y'all get it soon... I know I'm not the only one who'd love this feature.

[mad-ady]

Sorry, I've been swamped at work and haven't had time to push my changes. I want to do more testing (some measurements aren't getting through and need some debugging), and also I want to add more data to influx to be easier to filter. That requires that I understand Grammar (and I hated grammar in school) :P But I'm getting there. I need to finish a project that requires these changes, so I have my motivation :)

On Thu, Jan 30, 2020 at 8:09 PM sgroom-lb notifications@github.com wrote:

Looking forward to this... I've used smokeping for ever but this would allow me to do some remote polling and push it back to my tig stack @ our hq location.

Hope y'all get it soon... I know I'm not the only one who'd love this feature.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=ABAO6KSLAXKODJSGEYPQ6HLRAMJWBA5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKL64BQ#issuecomment-580382214, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KWZ3FSAL2Q5VAPRVLLRAMJWBANCNFSM4I6AUBOA .

[mad-ady]

Ok, I've started playing with this a bit and sent over the current implementation as a PR (https://github.com/oetiker/SmokePing/pull/215). It works, but it's not done yet. I still have to add support for dynamic configuration items, so that filtering on the influxdb/grafana side is effective.

So, let's review the problem... I'd like to be able to define targets that have configuration items in the form (as a regex) influx_.+ so that I can store arbitrary data (which is static) in influxdb with the goal of using it for filtering later. Some example configuration:

+ Hosts_in_region_A
menu = Hosts in region A
title = Hosts in region A

++ client1
host = 192.168.1.1
probe = fping
influx_download = 100Mbps
influx_upload = 10Mbps
influx_service_type = FTTB

++ client2
host = 22.22.222.22
probe = fping
influx_download = 20Mbps
influx_upload = 5 Mbps
influx_service_type = DSL

When doing the fping measurements and submitting data to influxdb, the code would add automatically the tags download=100Mbps, upload=10Mbps and service_type=FTTB, so you could add filters to see only FTTB measurements easily in Grafana.

As stated in the first post, currently Smokeping complains that it doesn't understand the influx_download (and others) configuration items.

So - I went around and tinkered with the Grammar building (without really understanding what's going on under the hood - it's a lot of code and references...). But, I found that if I add:

push @targetvars, "influx_test1";

in get_parser() in the Targets section, under the _dyn sub, at line 3674, right after my @targetvars = sort keys %$targetvars;, in this case, I am allowed to add a influx_test1 = ABCD config entry to my probes (it behaves like it's optional, though I haven't tested to see if the value is inherited or not).

So - I need to add something to targetvars, but this something needs to be tested by a regex, not compared exactly (like it is now). So influx_.+ doesn't work the way I'm hoping it would.

The lazy approach would be to hardcode influx_tag1, influx_tag2, ..., influx_tagN and simply use that instead, but it makes the configuration and influxdb/grafana graphs less readable/maintainable (who can remember that tag5 is customer bandwidth and tag7 is customer id?).

So, I'd like to kindly ask for help again! How can I make a Target configuration name be validated against a regex, instead of doing an exact match?

Thanks!

[oetiker]

I would suggest to create a new subsection ... +++ INFLUX_TAGS and then allow arbitrary key value pairs below that. for regex matching on keys you would have to use the syntax /influx_\S+/ as a definition for the variable.

[mad-ady]

I will try, thanks. Wouldn't that add a new menu level? Or complain there is no host declaration for the probe?

[oetiker]

I think if INFLUX_TAGS is added first then it should work fine ...

[mad-ady]

@oetiker: I'm seeing something weird in my git fork - it reports my fork is "This branch is 29 commits ahead, 28 commits behind oetiker:master.". Is it because you moved the commits to the 2.7.3 branch?

[oetiker]

this was merged with master (https://github.com/oetiker/SmokePing/pull/215) not with 2.73 afaik ... don't know about the state of your local checkout

[mad-ady]

Ok, sorry to bother you again, but due to time pressure, I took the easy approach and hardcoded some influx options on my branch: https://github.com/mad-ady/SmokePing/commit/426640ab47314afef43ab2ceb57622e704e30893 This works fine, on any level (e.g. ++, +++, ++++), but only for the first probe (or the FPing probe, it gets loaded/parsed first). Any idea why this happens?

[oetiker]

yes the problem is that you are messing with the data structures in a way that does not lead to predictable results ...

[mad-ady]

Fair enough :D

Plan C is to modify each plugin and add the influx_ parameters as optional. I expect that would work for my case. However, I still don't like that I don't understand how the Grammar module works/configuration is parsed. Is there a "development" example/tutorial that could help me better understand what is going on behind the scenes when the config file is parsed? Somewhere where I can enable all the debugging and follow the program flow? Surely you must have needed something like this during Grammar development...

Thanks

On Sun, Mar 8, 2020 at 12:58 PM Tobias Oetiker notifications@github.com wrote:

yes the problem is that you are messing with the data structures in a way that does not lead to predictable results ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=ABAO6KTXJDTW74TGBMIG7PDRGN24XA5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOESYQA#issuecomment-596192320, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KWVWIYIINHF4NNN53LRGN24XANCNFSM4I6AUBOA .

[oetiker]

Hi Adrian

there is documentation :-) [ https://metacpan.org/pod/Config::Grammar | https://metacpan.org/pod/Config::Grammar ]

cheers tobi

----- On Mar 10, 2020, at 8:35 AM, Adrian Popa notifications@github.com wrote:

Fair enough :D

Plan C is to modify each plugin and add the influx_ parameters as optional. I expect that would work for my case. However, I still don't like that I don't understand how the Grammar module works/configuration is parsed. Is there a "development" example/tutorial that could help me better understand what is going on behind the scenes when the config file is parsed? Somewhere where I can enable all the debugging and follow the program flow? Surely you must have needed something like this during Grammar development...

Thanks

On Sun, Mar 8, 2020 at 12:58 PM Tobias Oetiker notifications@github.com wrote:

yes the problem is that you are messing with the data structures in a way that does not lead to predictable results ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=ABAO6KTXJDTW74TGBMIG7PDRGN24XA5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOESYQA#issuecomment-596192320, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KWVWIYIINHF4NNN53LRGN24XANCNFSM4I6AUBOA .

— You are receiving this because you were mentioned. Reply to this email directly, [ https://github.com/oetiker/SmokePing/issues/201?email_source=notifications&email_token=AADIZX4S4WQTAX52XBP7JB3RGXUVDA5CNFSM4I6AUBOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOKKQOY#issuecomment-596944955 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AADIZX66XULL2WLLPM26BYDRGXUVDANCNFSM4I6AUBOA | unsubscribe ] .

-- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch tobi@oetiker.ch +41 62 775 9902

[mad-ady]

Ok, I took another crack at it. Still trying to go around Config::Grammar. Maybe bad memories from school put me off with working with Grammar :) So - I managed to get my way with some hardcoded optional target variables that get inherited by all (or most?) probes. I basically added the fields I need to lib/Smokeping/probes/base.pm inside targetvars: https://github.com/oetiker/SmokePing/commit/2b03c1c16733268f2f47beed6dd9b67d4a5ae30c

I realize this is still a hack (one that works for me, this time). My question to Tobi is - would you consider pulling it if I make the parameters loadable from some configuration file? Or will you accept only a proper Grammar-based implementation of these optional probe parameters?

For interested parties, I've packed my branch in this docker hub repo (I have images for x86, armhf, arm64): https://hub.docker.com/repository/docker/madady/smokeping-influx

The influxdb configuration section remains the same, the stuff I added allows me to define something like this:

+++ target1
host = 8.8.8.8
influx_connection_type = wired
influx_location = Location 1
influx_service = FTTH
influx_upload = 100
influx_download = 100

I will play some more with some Grafana graphs for my data and maybe share something with you (a proper how-to).

[meleschi]

@mad-ady I'm interested in the changes you've made to smokeping for it to write to influxdb. Is there anything I can do to help you test out your changes?

[mad-ady]

Sure, if you want the docker version, you can grab the latest build from here: https://hub.docker.com/repository/docker/madady/smokeping-influx I really need to work on documentation - sorry, I've been swamped. But I'll do it. For now, check out the docker-compose.yaml examples here for how to run the image: https://github.com/mad-ady/docker-smokeping. Inside the configuration file you will need to add a new section:

*** InfluxDB ***

host = 192.168.12.22
database = smokeping
timeout = 10
port = 8086

Once you start smokeping successfully you should see new measurements in that database. Note that currently only some tags are exported and I added a hack to include some custom tags that need to be defined at probe level, like this:

++ aldebaran
host = 192.168.1.5
influx_connection_type = wired
influx_location = Home
influx_service = FTTH
influx_upload = 300
influx_download = 300
influx_service_id = 192949320
influx_modem = RT56U

This part is subject to change in the future. Let us know how it goes and how your Grafana dashboards look like

On Mon, Mar 30, 2020 at 8:02 PM meleschi notifications@github.com wrote:

@mad-ady https://github.com/mad-ady I'm interested in the changes you've made to smokeping for it to write to influxdb. Is there anything I can do to help you test out your changes?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-606121485, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KULOFLINQ4EWG2PRMLRKDGBTANCNFSM4I6AUBOA .

[Shardy153]

Hi @mad-ady , Please add documentation on how to run the docker container.

I tried running the container with this command but it failed.

root@ip-10-131-56-10:/home/ubuntu/SmokePing# docker run madady/smokeping-influx
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_set_tz.sh...
*** /etc/my_init.d/01_set_tz.sh failed with status 1

*** Killing all processes...

[mad-ady]

I made examples on the github page: https://github.com/mad-ady/docker-smokeping There are also docker-compose examples in the smokeping-master-compose and smokeping-slave compose folders that you can edit to suit your needs. You can open an issue on my github in case you have problems, so that we don't go too much offtopic here.

I'll post back some Grafana examples based on data exported by Smokeping to influxdb...

On Fri, Jun 12, 2020 at 10:41 AM Shardy153 notifications@github.com wrote:

Hi, Please add documentation on how to run the docker container.

I tried running the container this command but it failed. `root@ip-10-131-56-10:/home/ubuntu/SmokePing# docker run madady/smokeping-influx Running /etc/my_init.d/00_regen_ssh_host_keys.sh... Running /etc/my_init.d/01_set_tz.sh... *** /etc/my_init.d/01_set_tz.sh failed with status 1

*** Killing all processes...`

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-643123963, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KU7V56FQRX2VLYJDGLRWHL4NANCNFSM4I6AUBOA .

[Shardy153]

I made examples on the github page: https://github.com/mad-ady/docker-smokeping There are also docker-compose examples in the smokeping-master-compose and smokeping-slave compose folders that you can edit to suit your needs. You can open an issue on my github in case you have problems, so that we don't go too much offtopic here. I'll post back some Grafana examples based on data exported by Smokeping to influxdb... … On Fri, Jun 12, 2020 at 10:41 AM Shardy153 @.> wrote: Hi, Please add documentation on how to run the docker container. I tried running the container this command but it failed. @.:/home/ubuntu/SmokePing# docker run madady/smokeping-influx Running /etc/my_init.d/00_regen_ssh_host_keys.sh... Running /etc/my_init.d/01_set_tz.sh... /etc/my_init.d/01_set_tz.sh failed with status 1 Killing all processes...` — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#201 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KU7V56FQRX2VLYJDGLRWHL4NANCNFSM4I6AUBOA .

Thanks for the quick reply

[mad-ady]

Ok, I've made time to clean things up and made a "how-to" post on my blog: http://adrianpopagh.blogspot.com/2020/06/smokeping-influxdb-export-docker-slaves.html It handles running the docker version of smokeping, exporting data to influxdb and setting up the Grafana dashboards.

I'd like people to read it and send feedback if something is missing or wrong.

Regarding Grafana - here are the generic dashboards I've created (links in the blog above): https://imgur.com/a/yPSwTFU

I'm also experimenting with different display methods - like status map, or a diagram, but those will be custom to my needs.

Overall I'm happy I was able to export data to influxdb, but I'm a bit disappointed with Grafana's variable support (variables are evaluated at dashboard load, not at graph load time). I'm unable to set some titles to my liking, so in the end I'll be generating the dashboards programmatically. :(

Regarding this PR, I'd like @oetiker 's opinion on how to close it. Since I made most of my commits I experimented with some things and apart from extra attributes that may be needed in influxdb which I hacked together, there are some fixes I'd like to be cherry-picked from my fork. Namely: https://github.com/mad-ady/SmokePing/commit/059cdb767b7fc2ae02d0f74ccc49e0da8df671cb https://github.com/mad-ady/SmokePing/commit/1d614ea5abeecc1a996cab0d73dbe1111a21ed97

https://github.com/mad-ady/SmokePing/commit/f1c5f4c50b0db6430ff07c72524975312e033831 (maybe, but still a hack?)

I have no idea how to send only these specific commits for inclusion. Thanks!

[oetiker]

@mad-ady to solve this, checkout master, make a new branch and then cherry-pick the commits you would like in your PR to the new branch. Finally, push the branch and PR so that I can review.

[Shardy153]

Hi, @mad-ady I tried the following this blog (http://adrianpopagh.blogspot.com/2020/06/smokeping-influxdb-export-docker-slaves.html) but while creating Smokeping master container I am getting the following error.

<13>Jun 15 10:56:15 ./run: Running smokeping as master
Jun 15 10:56:15 smokeping-master ./run: Running smokeping as master
ERROR: /config/Slaves, line 2: File '/opt/smokeping/etc/smokeping_secrets' does not exist

Could you please tell me how to fix this?

[Shardy153]

Hi, @mad-ady I also tried to run a docker container using the following command.

docker run -d \
    --name smokeping-master \
    --hostname smokeping \
    --net ping \
    -p 9500:80 \
    -e PUID=0 -e PGID=0 \
    -e TZ="UTC +0000" \
    -e SMOKEPING_EXTRA_FLAGS="--debug-daemon" \
    -v /home/ubuntu/smokeping/smokeping_secrets:/etc/smokeping/smokeping_secrets \
    -v /home/ubuntu/smokeping/data:/data \
    -v /home/ubuntu/smokeping/config:/config \
    madady/smokeping-influx:latest

Though the container was running, when I tried accessing the dashboard from the browser. http://MACHINE-IP:9500/smokeping/smokeping.cgi I am getting 404 as a response.

[mad-ady]

Try http://:9500/smokeping/smokeping.fcgi

On Mon, Jun 15, 2020 at 11:23 AM Shardy153 notifications@github.com wrote:

Hi, @mad-ady https://github.com/mad-ady I also tried to run a docker container using the following command.

docker run -d \ --name smokeping-master \ --hostname smokeping \ --net ping \ -p 9500:80 \ -e PUID=0 -e PGID=0 \ -e TZ="UTC +0000" \ -e SMOKEPING_EXTRA_FLAGS="--debug-daemon" \ -v /home/ubuntu/smokeping/smokeping_secrets:/etc/smokeping/smokeping_secrets \ -v /home/ubuntu/smokeping/data:/data \ -v /home/ubuntu/smokeping/config:/config \ madady/smokeping-influx:latest

Though the container was running, when I tried accessing the dashboard from the browser. http://:9500/smokeping/smokeping.cgi I am getting 404 as a response.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-643980686, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KUAZC3EMS5DRDAG4ZTRWXK7XANCNFSM4I6AUBOA .

[Shardy153]

Try http://:9500/smokeping/smokeping.fcgi … On Mon, Jun 15, 2020 at 11:23 AM Shardy153 @.***> wrote: Hi, @mad-ady https://github.com/mad-ady I also tried to run a docker container using the following command. docker run -d \ --name smokeping-master \ --hostname smokeping \ --net ping \ -p 9500:80 \ -e PUID=0 -e PGID=0 \ -e TZ="UTC +0000" \ -e SMOKEPING_EXTRA_FLAGS="--debug-daemon" \ -v /home/ubuntu/smokeping/smokeping_secrets:/etc/smokeping/smokeping_secrets \ -v /home/ubuntu/smokeping/data:/data \ -v /home/ubuntu/smokeping/config:/config \ madady/smokeping-influx:latest Though the container was running, when I tried accessing the dashboard from the browser. http://:9500/smokeping/smokeping.cgi I am getting 404 as a response. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#201 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KUAZC3EMS5DRDAG4ZTRWXK7XANCNFSM4I6AUBOA .

It worked, thanks. Please update the Github Readme as well.

[mad-ady]

Thanks for the heads up. I just did.

On Mon, Jun 15, 2020 at 11:34 AM Shardy153 notifications@github.com wrote:

Try http://:9500/smokeping/smokeping.fcgi … <#m7578931876103385987> On Mon, Jun 15, 2020 at 11:23 AM Shardy153 @.***> wrote: Hi, @mad-ady https://github.com/mad-ady https://github.com/mad-ady I also tried to run a docker container using the following command. docker run -d \ --name smokeping-master \ --hostname smokeping \ --net ping \ -p 9500:80 \ -e PUID=0 -e PGID=0 \ -e TZ="UTC +0000" \ -e SMOKEPING_EXTRA_FLAGS="--debug-daemon" \ -v /home/ubuntu/smokeping/smokeping_secrets:/etc/smokeping/smokeping_secrets \ -v /home/ubuntu/smokeping/data:/data \ -v /home/ubuntu/smokeping/config:/config \ madady/smokeping-influx:latest Though the container was running, when I tried accessing the dashboard from the browser. http://:9500/smokeping/smokeping.cgi I am getting 404 as a response. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#201 (comment) https://github.com/oetiker/SmokePing/issues/201#issuecomment-643980686>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KUAZC3EMS5DRDAG4ZTRWXK7XANCNFSM4I6AUBOA .

It worked, thanks. Please update the Github Readme as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-643986312, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KRK6PUJZWRPXKDWVZTRWXMIHANCNFSM4I6AUBOA .

[mad-ady]

Just found out a bug in my implementation. I am writing data to influxdb with the timestamps from the master system. However, in case a slave gets disconnected from the master, but still takes measurements and reports them later, it will cause all the measurements to be reported with almost the same timestamp. I've checked and the rrds are written correctly. I'll need to add the timestamp from the slave as well - thank Tobi it's sent (I thought it wasn't).

Here's an example measurement with past data (note the consecutive timestamps):

2020-06-17T05:14:36.273Z 3    100          IQ-01-lan www.wikipedia.org
2020-06-17T05:14:36.283Z 3    100          IQ-01-lan www.wikipedia.org
2020-06-17T05:14:36.293Z 3    100          IQ-01-lan www.wikipedia.org
2020-06-17T05:14:36.303Z 3    100          IQ-01-lan www.wikipedia.org
2020-06-17T05:14:36.313Z 3    100          IQ-01-lan www.wikipedia.org
2020-06-17T05:19:36.91Z  3    100          IQ-01-lan www.wikipedia.org
2020-06-17T05:24:34.999Z 0    0            IQ-01-lan www.wikipedia.org
2020-06-17T05:29:34.934Z 0    0            IQ-01-lan www.wikipedia.org
2020-06-17T05:34:35.816Z 0    0            IQ-01-lan www.wikipedia.org

[mad-ady]

I've made a fix for the problem above and rebuilt the docker images. If you encounter the problem make sure to delete and pull the new images.

[PuckStar]

Time for some progress report. TL;DR: it works, but needs polishing.

Managed to implement pushing data to influxdb. Also I docker-ized a smokeping master/slave install based on my branch. Data gets written to influxdb fine, but I need to expose more keys so that filtering would be easier. I'm still lost when it comes to Config::Grammar - @oetiker, maybe you can give me a tip in the right direction (exposing optional influx_* key/value pairs in any configuration section). I plan on sending a PR after merging these changes as well

How to test current changes: build smokeping from my branch here: https://github.com/mad-ady/SmokePing, or get the docker images from here (it will download the correct image for your architecture): https://hub.docker.com/repository/docker/madady/smokeping-influx. On this page I have some test docker-compose scripts to set up a master and a slave instance: https://github.com/mad-ady/docker-smokeping I need to work more on documentation...

Now, to enable influx export, you'll need to add the following section to your smokeping config:

*** InfluxDB ***

host = 192.168.100.100
database = smokeping
timeout = 10
port = 8086

You need to create a database like so (assumes influx is running in a container, like everything else does this decade):

root@iq-management:~# docker exec -it influxdb bash  
root@5d53c9e39e75:/# influx
Connected to http://localhost:8086 version 1.7.6
InfluxDB shell version: 1.7.6
Enter an InfluxQL query
> create database smokeping;
> exit
root@5d53c9e39e75:/# exit

To check that data is inserted in influxdb, you can run queries like:

select * from FPing

I also set up a minimal Grafana dashboard that uses influx data. It's generic enough to get you started, but needs polishing (like suppressing lines where data is missing, but connecting adjacent dots). Here is the template: https://pastebin.com/qrHNXExZ

Here is how it looks like:

I'm showing only min/max/average and loss percent, though all measurements (ping*) are stored in influx for further processing.

Suggestions and testing are more than welcome!

I'd really like to try this (pushing the date to influxdb) but I already installed smokeping using this guide on a pi3. Do I have to start all over to use your version? Or can I just somehow install it over the one I already have? And what are the precise steps? As I am no linux (or docker) expert.

[mad-ady]

You can simply replace these files from your smokeping folder with the ones changed on my repo: https://github.com/oetiker/SmokePing/compare/master...mad-ady:master, and also install the extra perl modules and you should be good to go.

On Sat, Jun 27, 2020, 01:11 PuckStar notifications@github.com wrote:

Time for some progress report. TL;DR: it works, but needs polishing.

Managed to implement pushing data to influxdb. Also I docker-ized a smokeping master/slave install based on my branch. Data gets written to influxdb fine, but I need to expose more keys so that filtering would be easier. I'm still lost when it comes to Config::Grammar - @oetiker https://github.com/oetiker, maybe you can give me a tip in the right direction (exposing optional influx_* key/value pairs in any configuration section). I plan on sending a PR after merging these changes as well

How to test current changes: build smokeping from my branch here: https://github.com/mad-ady/SmokePing, or get the docker images from here (it will download the correct image for your architecture): https://hub.docker.com/repository/docker/madady/smokeping-influx. On this page I have some test docker-compose scripts to set up a master and a slave instance: https://github.com/mad-ady/docker-smokeping I need to work more on documentation...

Now, to enable influx export, you'll need to add the following section to your smokeping config:

InfluxDB

host = 192.168.100.100 database = smokeping timeout = 10 port = 8086

You need to create a database like so (assumes influx is running in a container, like everything else does this decade):

root@iq-management:~# docker exec -it influxdb bash root@5d53c9e39e75:/# influx Connected to http://localhost:8086 version 1.7.6 InfluxDB shell version: 1.7.6 Enter an InfluxQL query

create database smokeping; exit root@5d53c9e39e75:/# exit

To check that data is inserted in influxdb, you can run queries like:

select * from FPing

I also set up a minimal Grafana dashboard that uses influx data. It's generic enough to get you started, but needs polishing (like suppressing lines where data is missing, but connecting adjacent dots). Here is the template: https://pastebin.com/qrHNXExZ

Here is how it looks like: [image: alt text] https://camo.githubusercontent.com/a3112e9707ef61fc063273ab78e3f25af6007f67/68747470733a2f2f696d6775722e636f6d2f715146784d6a712e706e67

I'm showing only min/max/average and loss percent, though all measurements (ping*) are stored in influx for further processing.

Suggestions and testing are more than welcome!

I'd really like to try this (pushing the date to influxdb) but I already installed smokeping using this guide https://community.spiceworks.com/how_to/96949-raspberrypi-rpi-smokeping-setup-and-configuration on a pi3. Do I have to start all over to use your version? Or can I just somehow install it over the one I already have? Mind you I am no linux (or docker) expert.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-650426148, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KTRBKMRNBLWJOXC3KLRYUMKPANCNFSM4I6AUBOA .

[PuckStar]

You can simply replace these files from your smokeping folder with the ones changed on my repo: master...mad-ady:master, and also install the extra perl modules and you should be good to go.

I think I'll manage to overwrite those files but can you tell me the command to install the perl modules?

[mad-ady]

Something like sudo perl -MCPAN -e 'install InfluxDB::HTTP', for each of the modules.

On Thu, Jul 2, 2020 at 9:54 PM PuckStar notifications@github.com wrote:

You can simply replace these files from your smokeping folder with the ones changed on my repo: master...mad-ady:master https://github.com/oetiker/SmokePing/compare/master...mad-ady:master, and also install the extra perl modules and you should be good to go.

I think I'll manage to overwrite those files but can you tell me the command to install the perl modules?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-653169627, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KXZPLFODLOVYVJ33KDRZTJV3ANCNFSM4I6AUBOA .

[PuckStar]

Something like sudo perl -MCPAN -e 'install InfluxDB::HTTP', for each of the modules. … On Thu, Jul 2, 2020 at 9:54 PM PuckStar @.***> wrote: You can simply replace these files from your smokeping folder with the ones changed on my repo: master...mad-ady:master <master...mad-ady:master>, and also install the extra perl modules and you should be good to go. I think I'll manage to overwrite those files but can you tell me the command to install the perl modules? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#201 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KXZPLFODLOVYVJ33KDRZTJV3ANCNFSM4I6AUBOA .

I managed (I think) to install it with sude cpan InfluxDB::HTTP I didn't know if I only needed to install that so I actually went through all those mentioned in the file :).

Now I thought it was gonna be easy to replace 3 the files that you mention but I looked almost everyone on the pi3 but can't find those files. In the smokeping folder I see some files but none are for instance smokeping.pm.

Sorry for my probably noobish questions. If you're fed up with it just let me know :).

[mad-ady]

Note that it's case sensitive: root@smokeping-master:/# ls -l /opt/smokeping/lib/Smokeping.pm -rw-r--r-- 1 root root 195274 Jun 19 11:23 /opt/smokeping/lib/Smokeping.pm

Run a find / -name Smokeping.pm

On Fri, Jul 3, 2020 at 11:11 AM PuckStar notifications@github.com wrote:

Something like sudo perl -MCPAN -e 'install InfluxDB::HTTP', for each of the modules. … <#m-5103328622612348406> On Thu, Jul 2, 2020 at 9:54 PM PuckStar @.***> wrote: You can simply replace these files from your smokeping folder with the ones changed on my repo: master...mad-ady:master <master...mad-ady:master https://github.com/oetiker/SmokePing/compare/master...mad-ady:master>, and also install the extra perl modules and you should be good to go. I think I'll manage to overwrite those files but can you tell me the command to install the perl modules? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#201 (comment) https://github.com/oetiker/SmokePing/issues/201#issuecomment-653169627>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KXZPLFODLOVYVJ33KDRZTJV3ANCNFSM4I6AUBOA .

I managed (I think) to install it with sude cpan InfluxDB::HTTP I didn't know if I only needed to install that so I actually went through all those mentioned in the file :).

Now I thought it was gonna be easy to replace 3 the files that you mention but I looked almost everyone on the pi3 but can't find those files. In the smokeping folder I see some files but none are for instance smokeping.pm.

Sorry for my probably noobish questions. If you're fed up with it just let me know :).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-653415453, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KXDP35PTJSRNM2IP3LRZWHC3ANCNFSM4I6AUBOA .

[PuckStar]

Note that it's case sensitive: root@smokeping-master:/# ls -l /opt/smokeping/lib/Smokeping.pm -rw-r--r-- 1 root root 195274 Jun 19 11:23 /opt/smokeping/lib/Smokeping.pm Run a find / -name Smokeping.pm

I got it working!!!! https://i.imgur.com/d5d4X0i.png Thanks a lot for your patience and help!

[xrayou]

@mad-ady Hi, does it support slave node to write rrd data to influxdb, I have done with Master and Slave, and it seem works fine as bellow; but in in the influxdb, I can only see the master data, can you help to see what is the problem, thanks !

[mad-ady]

Data for the slave should be written in influxdb as well. Look for its name (JZNC in your case). Data gets written to influx by the master instance (only master communicates with influxdb), but would write when data from the slaves is reported back. Check the smokeping log as well, and maybe the influxdb log for errors (maybe some data types are different and influxdb refuses to store the data).

[mad-ady]

Hello all. I've made a new PR with a few of my fixes to SmokePing master: https://github.com/oetiker/SmokePing/pull/257. After applying it you no longer need to use my fork.

[Strykar]

Hello all. I've made a new PR with a few of my fixes to SmokePing master: #257. After applying it you no longer need to use my fork.

Sweetness, nice work Ady!

[JimKlapwijk]

Hi @mad-ady. Thank you very much for your hard work, appreciate it! I got it working on my LXC container. Now I need a way to automate it via Saltstack, but we'll get there.

I just have one issue, I run InfluxDB on HTTPS. and I entered as host https://influxdb.example.com. It still gives errors:

Jan 26 20:14:14 smokeping smokeping[269239]: Error inserting measurement into influxdb: Error executing write: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "Can't connect to [ht...") at /usr/local/share/perl/5.30.0/InfluxDB/HTTP.pm line 161.#012 for $VAR1 = [#012          'FPing,host=192.168.1.248,loss=0,loss_percent=0,path=/home/wired/prometheus,slave=master,title=192.168.1.248\\ -\\ prometheus loss=0i,loss_percent=0i,max=6.000000e-05,median=6.000000e-05,min=3.000000e-05,ping1=3.000000e-05,ping10=6.000000e-05,ping11=6.000000e-05,ping12=6.000000e-05,ping13=6.000000e-05,ping14=6.000000e-05,ping15=6.000000e-05,ping16=6.000000e-05,ping17=6.000000e-05,ping18=6.000000e-05,ping19=6.000000e-05,ping2=4.000000e-05,ping20=6.000000e-05,ping3=4.000000e-05,ping4=5.000000e-05,ping5=5.000000e-05,ping6=5.000000e-05,ping7=5.000000e-05,ping8=6.000000e-05,ping9=6.000000e-05 1611688454000'#012        ];

Does it support HTTPS, if yes, how can I enable it? Again, thank you very much!!

[mad-ady]

I haven't tested HTTPS. It probably doesn't work out of the box, but I'm pretty sure it could be added without too much trouble, similarly to how HTTP auth was added, by getting the underlying LWP object that handles the connection...

https://metacpan.org/pod/InfluxDB::HTTP

Something like this: https://github.com/mad-ady/SmokePing/commit/c2034a858ea53afc485157773f138c653edb84ea

See this for documentation: https://metacpan.org/pod/LWP::UserAgent. Most likely you'll need to allow self-signed certificates...

Update: upon further digging in InfluxDB::HTTP's code I can see that the scheme is hardcoded to http:

sub _get_influxdb_http_api_uri {
    my ($self, $endpoint) = @_;

    die "Missing argument 'endpoint'" if !$endpoint;

    my $uri = URI->new();

    $uri->scheme('http');

You'd need to send a PR to InfluxDB::HTTP to support https as well (via a configurable default), and a PR to me (or @oetiker directly) to support a new scheme config option. So, not impossible, just a bit of work. Let me know if you're not into it and I may try to do it when time permits...

On Tue, Jan 26, 2021 at 9:16 PM JimKlapwijk notifications@github.com wrote:

Hi @mad-ady https://github.com/mad-ady. Thank you very much for your hard work, appreciate it! I got it working on my LXC container. Now I need a way to automate it via Saltstack, but we'll get there.

I just have one issue, I run InfluxDB on HTTPS. and I entered as host https://influxdb.example.com. It still gives errors:

Jan 26 20:14:14 smokeping smokeping[269239]: Error inserting measurement into influxdb: Error executing write: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "Can't connect to [ht...") at /usr/local/share/perl/5.30.0/InfluxDB/HTTP.pm line 161.#012 for $VAR1 = [#012 'FPing,host=192.168.1.248,loss=0,loss_percent=0,path=/home/wired/prometheus,slave=master,title=192.168.1.248\ -\ prometheus loss=0i,loss_percent=0i,max=6.000000e-05,median=6.000000e-05,min=3.000000e-05,ping1=3.000000e-05,ping10=6.000000e-05,ping11=6.000000e-05,ping12=6.000000e-05,ping13=6.000000e-05,ping14=6.000000e-05,ping15=6.000000e-05,ping16=6.000000e-05,ping17=6.000000e-05,ping18=6.000000e-05,ping19=6.000000e-05,ping2=4.000000e-05,ping20=6.000000e-05,ping3=4.000000e-05,ping4=5.000000e-05,ping5=5.000000e-05,ping6=5.000000e-05,ping7=5.000000e-05,ping8=6.000000e-05,ping9=6.000000e-05 1611688454000'#012 ];

Does it support HTTPS, if yes, how can I enable it? Again, thank you very much!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oetiker/SmokePing/issues/201#issuecomment-767767310, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAO6KRES2EVI72CE2KJ2XDS34IILANCNFSM4I6AUBOA .