New Datasource Plugins

[bhardgrove]

Is there any interest in me adding datasource plugins to some newer modern DB structures?

Support for a REST API, elasticsearch, or InfluxDB?

I have wrote a number of datasource plugins for various systems and can share them.

[howardjones]

Oh, definitely! That's been on my todo list for a while now...

It would be interesting to get your thoughts on how possible it is to add data source pickers in the editor for those APIs too... I want to replace the current picker that only understands cacti with something else that uses plugins to allow you to browse other data source. Obviously the datasource would need to know how to supply the browsing info.

[sjthespian]

I'm trying to shut down my grafana server in favor of influxDB, so I'll add a vote for influxDB support! There is an older datasource plugin floating around on GitHub, but it doesn't seem to work with influxDB 1.x and doesn't work with more advanced select statements.

As an example, I need to be able to query something like:

SELECT derivative(mean("ifHCInOctets"), 1s) FROM "ifXTable" WHERE "hostname" = 'switch.example.com' AND "ifName" = 'Gi2/0/48' AND time > now() - 1m GROUP BY time(30s)

[howardjones]

So this is a wider subject I've been thinking about for a while. Several plugins are more like "meta-plugins", where you might want to define different queries as named target prefixes somehow, and then use those prefixes... I'm thinking of anything that uses complex queries like this. It would be much better to have ONE "database query" plugin and one "influxdb query" plugin, and have some (central? shared?) configuration that you'd add things something like:

avginbytes:*:*    SELECT derivative(mean("ifHCInOctets"), 1s) FROM "ifXTable" WHERE "hostname" = '%s1' AND "ifName" = '%s2' AND time > now() - 1m GROUP BY time(30s)

but with something cleaner (and hopefully not regexps) to do the defining. Then you'd use TARGET avginbytes:switch.example.com:Gi2/0/48 in the map config. Same kind of thing for SQL queries.

It would avoid lots of datasource plugins which are identical except for the query, or lots of long query strings embedded in the map configs...

What do you think?

[bhardgrove]

There are a few ways that I think we could design this. What if we just used the datasource plugin to do a file_get_contents on an API call using a bunch of set commands.

SET ip 10.1.1.1 SET kpi traffic_in SET timePeriod 30s

Then you can use these fields to do an API call. That API is where the logic takes place and converts those values to a specific query. I have been writing this in NodeJS as it is much faster than PHP.

This would become select metric from table where time > $timePeriod etc.

On Mon, Apr 17, 2017 at 3:34 PM, Howard Jones notifications@github.com wrote:

So this is a wider subject I've been thinking about for a while. Several plugins are more like "meta-plugins", where you might want to define different queries as named target prefixes somehow, and then use those prefixes... I'm thinking of anything that uses complex queries like this. It would be much better to have ONE "database query" plugin and one "influxdb query" plugin, and have some (central? shared?) configuration that you'd add things something like:

avginbytes:: SELECT derivative(mean("ifHCInOctets"), 1s) FROM "ifXTable" WHERE "hostname" = '%s1' AND "ifName" = '%s2' AND time > now() - 1m GROUP BY time(30s)

but with something cleaner (and hopefully not regexps) to do the defining. Then you'd use TARGET avginbytes:switch.example.com:Gi2/0/48 in the map config. Same kind of thing for SQL queries.

It would avoid lots of datasource plugins which are identical except for the query, or lots of long query strings embedded in the map configs...

What do you think?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/howardjones/network-weathermap/issues/59#issuecomment-294615281, or mute the thread https://github.com/notifications/unsubscribe-auth/ALvX4GnRd3Gl2eTOo3wyMVB7JzCsQBssks5rw-jigaJpZM4MFTYY .

[sjthespian]

InfluxDB is flexible (for lack of a better word) enough that Howie's suggestion may be the only way to go. Some of the queries I have ended up with in Grafana are somewhat convoluted, and I think the ability to define a query with parameters might be the most flexible way of dealing with it.

Building a plugin for ElasticSearch could fall along similar lines, as you just need a way to specify a JSON structure with parameters to populate it.

[howardjones]

Indeed - and also a nicer SNMP plugin that had canned queries rather than obnoxiously long target strings.