search

daq-tools / kotori

A flexible data historian based on InfluxDB, Grafana, MQTT, and more. Free, open, simple.
https://getkotori.org/
GNU Affero General Public License v3.0
109 stars 17 forks source link

Use specific time precision when sending metrics to InfluxDB using UDP #29

Open amotl opened 4 years ago

amotl commented 4 years ago

Together with @adrianlzt, we discussed the option to send metrics to InfluxDB using UDP for high frequency data acquisition within #28.

amotl commented 4 years ago

[...] I also would like to point out that you can configure Kotori to use the UDP protocol to talk to InfluxDB (see [1] and the use_udp option within [2]) in order to improve the acquisition throughput by getting rid of the overhead coming from HTTP.

However, this feature is not obvious, poorly documented and a bit weird to configure, see [3-5]. If you would like to use that feature to increase your data rate without wasting resources, I will see what I can do to improve the current situation.

The main weirdness of the implementation has its background that the database will not get created automatically when data is acquired through UDP. Only the HTTP interface to InfluxDB will provide the appropriate response status to create the database [6]. So, even when using the UDP data acquisition path, Kotori will have to use the HTTP interface for a single time.

[1] https://docs.influxdata.com/influxdb/v1.7/supported_protocols/udp/ [2] https://influxdb-python.readthedocs.io/en/latest/api-documentation.html#influxdbclient [3] https://getkotori.org/docs/gallery/luftdaten.info/index.html#id2 [4] https://github.com/daq-tools/kotori/blob/master/kotori/daq/storage/influx.py#L28-L29 [5] https://github.com/daq-tools/kotori/blob/master/kotori/daq/storage/influx.py#L39-L43 [6] https://github.com/daq-tools/kotori/blob/master/kotori/daq/storage/influx.py#L89-L94

amotl commented 4 years ago

@adrianlzt commented:

Yes, I saw that option, but, correct me if I'm wrong, but only second precision is available and I need ms.

amotl commented 4 years ago

Dear Adrián,

but only second precision is available and I need to express milliseconds

You might be correct on that important detail and now I remember it when using it the other day. Regarding this issue, I found https://github.com/influxdata/influxdb/issues/841 and https://github.com/influxdata/influxdb/issues/8746 on the InfluxDB issue tracker.

The latter one refers to a precision options which could be used inside the [[udp]] section of the configuration file like

[[udp]]
  # ...

  # InfluxDB precision for timestamps on received points ("" or "n", "u", "ms", "s", "m", "h")
  # precision = ""

So, if you are curious to check this, I will also be happy to learn about whether the precision = "" option actually works. Please note that in recent versions of InfluxDB, they might be called ns, us, ms, s, m, h already.

With kind regards, Andreas.

References

At the InfluxDB issue tracker, there are quite a few issues revolving around this topic: