search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.48k stars 5.54k forks source link

Integration of vFlow or Goflow Netflow/IPFIX collector module in a Telegraf input plugin #8425

Closed door7302 closed 1 year ago

door7302 commented 3 years ago

Feature Request

NetFlow/IPFIX collector as Telegraf's input plugin

Proposal:

There are several nice Go Projects that provide reliable and scalable solutions to collect Netflox/IPFIX sample information from network devices.

It would very useful to "backport" the " Netflow/IPFIX collector module" of one of this project in a dedicated Telegraf Input plugin.

Current behavior:

Desired behavior:

Use case:

Currently we collect traffic stats of our networking devices via the gnmi Telegraf plugin. We also collect in // Netflow/IPFIX with another tool and we wish to have a single collector solution to collect all data from our network. We love Telegraf solution and moreover we have already developed several Telegraf Processor Plugins to manipulate and enrich our data collected by telegraf input plugin and we wish to reuse them for Netflow/IPFIX sampled data.

sjwang90 commented 3 years ago

Currently we don't support binary (ex: protobuf) formats. There is a PR #3421 for support that may need some more progress.

I've opened some issues in goflow/vFlow repos to see if they're willing to support any other output formats in the meantime.

lspgn commented 3 years ago

GoFlow developer/maintainer here: I'm not familiar with Telegraf/Influx, but if we are dealing with time-series, you will need to do an aggregation before insertion (eg: summing all sFlow packets by a key like destination ASN, port and protocol) which unfortunately is out of scope of the decoder.

Alternative that I've seen used with Logstash, you can have GoFlow output the samples as JSON in stdout (not using Kafka+protobuf).

door7302 commented 3 years ago

Thank you for your interest of my request. From my side I will have a look if I could 'POC' some stuff to integrate a part of goflow code in an input plugin. I will share my tests. We will also need a plugin of "aggregation" to aggregate sample data before sending to "Timeseries" DB to avoid the explosion of cardinality :)

It's a nice challange I guess. As i'm not a "expert" in dev I will share my tests, sample codes and maybe it will help to integrate something more reliable latter :)

sspaink commented 1 year ago

@door7302 were you able to make any progress on the proof of concept?

telegraf-tiger[bot] commented 1 year ago

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!