search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.17k stars 5.52k forks source link

Adding a "first" and "last" measurement option to the BasicStats aggregator #11097

Open bikeymouse opened 2 years ago

bikeymouse commented 2 years ago

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Besides the current measurements, such as "min", "max" and "mean" I would like to have the "last" or "first" value out of each period being returned.

Current behavior:

This is not available currently.

Desired behavior:

Be able to specify that the "first" and/or "last" value out of all measurements in every "period" of seconds is returned out of a series of continuous measurements.

Note: the "Final" Aggregator does not work as required, as this only submits a " final" measurement if there are no measurements during a set timeout. This however does not work for continuous streams of measurements without interruptions/timeouts.

Use case:

I have measurements published on Mosquito that I wish to monitor in real-time. So the input to telegraf/Mosquitto must not be throttled. For analysis however the measurements could be reduced from 1 sample per second to once per minute.

With this new feature this could be done in memory/Telegraf before the data is written to Influx, which drastically reduces the number of writes to e.g. Influx.

powersj commented 2 years ago

Hi,

In your scenario, it sounds like you have one input (?) and two outputs: mosquito and influxdb? And you want all the data to continue going to mosquito as-is, but reduce only the amount going to influxdb?

Thanks

bikeymouse commented 2 years ago

Currently I have external systems putting the data onto Mosquito and I'm using Telegraf to get that into Influx. So Telegraf has 1 input: Mosquito, and 1 output: Influxdb.

I could reduce the data collection interval for Telegraf, but that still would not reduce the number of read measurements as Mosquito is queuing them until being read.

In future use I may use Telegraf as well to get the input onto Mosquito, so then indeed I would have a couple of inputs from which the data would have to go to mosquito as-is, but only the reduced amount going into influx, as you describe.

Reducing the data collection then is not even an option anymore as I would need to publish the data to Mosquito in real-time.

powersj commented 2 years ago

@srebhan @reimda, I am trying to find a previous issue to reference, but I did not think we guaranteed order in the data telegraf receives and sends? Is that still the case or have I mistaken this with something else? If it is, I am not sure this is something we could implement.

srebhan commented 2 years ago

@powersj: TL;DR: No there is no guarantee on the temporal order of the samples.

I don't have a ticket either, but the issue is that an order in the "time the data is received by telegraf" is not necessarily a global temporal order. Imagine an input that gathers data in temporal order and another input that occasionally outputs historic data (e.g. from some hours ago). Even if we guarantee "ordered by arrival time" (which I'm not even sure we do), in the aggregator the "old" data will just arrive somewhere in between the "current" data.

What you can do, is you order the batch you are currently looking at, sort by timestamp and then output "first"/"last". That will probably work for "many setups", but you still see above issues with others. So if we do it, please add a big fat warning to the README.

bikeymouse commented 2 years ago

Using the timestamps of the measurements, even if it would be “historical” data, would work fine for my use case! Also note that if one does this data reduction afterwards in Influx there would also be no other way then to sort on the timestamp.

srebhan commented 2 years ago

@bikeymouse do you want to give it a try and submit a PR? Feel free to ping me on it and we can work out the details...

bikeymouse commented 2 years ago

@srebhan I'm very sorry, I would if I could, but I'm not a developer so I'm afraid I can't do the coding myself, just test it functionally in my environment.

bikeymouse commented 1 year ago

Just a gentle reminder: anybody else also needing this functionality and/or able to make a PR for this? I'd think that this has many use cases as this is a great way to reduce the amount of data dat is sent to e.g. Influx, throttling this a close as possible to the source. I would be very happy to test ;-)