search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.69k stars 5.59k forks source link

Add the ability to indicate IPFIX templates within PEN Map #16169

Open vs0t opened 5 days ago

vs0t commented 5 days ago

Use Case

We are currently sending IPFIX data from a device. Initially, the device was sending data points directly, and our PEN map was able to correctly decode and process the data. However, since we've transitioned to using templates in our IPFIX configuration, we're encountering issues with decoding the data. The templates define the structure of the data, but Telegraf does not seem to process it correctly.

Our goal is to extend the current PEN map to support IPFIX templates, ensuring we can continue receiving and decoding the same data we were sending previously, but now as part of a structured template.

Expected behavior

The expected behavior is that within the PEN map csv, you can indicate a template ID, and within the template you can indicate the different ID's for the data being sent. The netflow plugin will be able to see when a template arrives, and use the template and its corresponding ID's within the PEN map to then decode and process the data as usual.

Actual behavior

With my previous PEN map, once we started sending it as a template, I stopped seeing data come into the bucket, and could see the errors within the logs. My PEN map contains a lot of data points, so its not an issue of me missing some ID's within the map.

Additional info

Nov 08 14:40:24 xxxx telegraf[152640]: 2024-11-08T19:40:24Z E! [inputs.netflow] Error in plugin: decoding message failed: IPFIX IPFIX [version:10 type:DataSet obsDomainId:0: templateId:699] DataSet: fields [unexpected EOF]; raw data: xxxx...

If this is something due to my error, please let me know so I can delete this. I don't meant to bother, just not sure if this is a feature that could be added, or if it's due to my errors.

vs0t commented 5 days ago

Ignore the log in the additional info, this was due to the configuration on the device. The question still stands on the ability to ingest templates, and parse them. These templates will contain data from multiple ONT's within them, looking for a way to utilize the templates to construct dashboards/queries that can show data from the mutliple ONT's within the template.