search

CESNET / ipfixcol2

High-performance NetFlow v5/v9 and IPFIX collector (RFC7011)
Other
123 stars 36 forks source link

Deduplication Feature in ipfixcol2 tool #96

Closed sreekanth-h closed 5 months ago

sreekanth-h commented 5 months ago

HI Team, We are planning to use IPFIXCol2 as a collector for our NetFlow collection as a replacement to our existing vendor tool. The current tool got the deduplication feature and would like to know this feature is there in ipfixcol2? Currently we are trying our POC's with UDP input plugin and json output plugin that send converted logs to other server with "send" function.

Thanks, Sree

Lukas955 commented 5 months ago

Hi,

thank you for using IPFIXcol2.

We have previously considered and developed initial attempts of deduplication functionality. However, it is not completely trivial both to implement and to correctly configure by users. In a nutshell, probes from different vendors typically have different outputs and potentially different key items identifying each flow. This could be configurable. However, if duplicate records are subsequently received from different probes, but e.g. may differ in packet/byte count or other (e.g. application) fields, which record to select or prefer? There are a number of non-trivial issues that need to be implemented and resolved in a very generic way.

In other words, we do not support this functionality at the moment. We may add it sometime in the future, but right now we don't even have plans to do so.

Lukas

sreekanth-h commented 5 months ago

Thanks Lukas. That makes sense. The deduplication feature needs to be considered case by case basis. Any opensource tools as suggestions you have to try and look.

Lukas955 commented 5 months ago

Hi,

Unfortunately, I don't know any tool that I can recommend for this feature.

Lukas

sreekanth-h commented 5 months ago

Thanks Lukas