search

KEGustafsson / SignalK-Telegraf-Influxdb-Grafana-in-docker

SignalK, Telegraf, Influxdb and Grafana in docker. Based on https://github.com/SignalK/signalk-server
3 stars 3 forks source link

upgrade influx to v2 #8

Open leeprevost opened 2 years ago

leeprevost commented 2 years ago

I and other users have successfully gotten the influxdb plugin to support the new v2 influx db.

https://github.com/tkurki/signalk-to-influxdb/issues/63

Also, there is work underway to upgrade the plug in to support the v2 client.

What would need to happen to move to influxdb oss version 2?

KEGustafsson commented 2 years ago

@leeprevost, I have had test docker-compose for Influxdb v2 for quite sometime already (dated late 2020). https://github.com/KEGustafsson/influxdbv2-test-docker

Would you like to test if this one would be working solution? I updated it to pointing latest Influxdb image, influxdb:latest.

There is also Telegraf instance reading SK vessel self http json and putting that data to Influxdb. If this is a working solution then I could merge changes here too.

To migrate data from v1.x to v2.3, check guide from here https://docs.influxdata.com/influxdb/v2.3/upgrade/v1-to-v2/

leeprevost commented 2 years ago

That sounds pretty cool.

On the Telegraf question, I've been using the signalk-to-influxdb plug in pulling in self data at 200ms intervals. My requirement is to do some light math on the high freq data (show things like temp change per meter traveled via derivative (waterTemp)/speedThroughWater to show tempChgPerMeter. I can then capture anomalies only, save to another bucket and then downsample the data to something like 1sec or even less.

I see you are using 60sec. Is that becaause Telegraf has to poll slowly? can it poll faster? Also., how does Telegraf put the data into the db? Is it all values into one record or breaks each key on self into separate measureements? If the latter, this is similar to how the plug-in works. I'm struggling to normalize them, join them, and then do the math like the above when measurements don't come in on same timestamps. I assume that is because of the N2k bus.

So, I've migrated and am now using influxdb2 but am still on the older plug-in.

If I follow you right, Telegraf allows you to avoid using a plug in at all? Or be independent of javascript an client versions?

KEGustafsson commented 2 years ago

Telegraf can accept wide range of intervals, so no problem to make it faster or slower. https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md#intervals

Input from http/https using JSON https://github.com/influxdata/telegraf/blob/master/plugins/inputs/http/README.md https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json

Output to Influxdb https://github.com/influxdata/telegraf/blob/master/plugins/outputs/influxdb_v2/README.md

KEGustafsson commented 2 years ago

It seams that old run_me script is not working, so Influxdb_v2 account need to be done via cli/web at the moment. I'll fix this later.

Anyway, I can see data coming in via Telegraf to bucket. My Flux language knowledge is so poor that I cannot do detailed queries at the moment with Grafana, but with Influxdb GUI I can see all http JSON data

KEGustafsson commented 2 years ago

Now script and docker-compose are fixed. https://github.com/KEGustafsson/influxdbv2-test-docker