search

CESNET / ipfixcol2

High-performance NetFlow v5/v9 and IPFIX collector (RFC7011)
Other
117 stars 35 forks source link

setting up record describing the structure of stream records (i.e. ipfix.entry) #90

Open Catwoolfii opened 5 months ago

Catwoolfii commented 5 months ago

Hello everybody!

I just tried this collector and it's just great! I have a problem with configuring json output, specifically: changing record types. I can't figure out how to apply the keys "ipfix.template" and "ipfix.optionsTemplate" in configuration file. Can anyone show me an example of how to set this up?

Lukas955 commented 5 months ago

Hi,

I'm not quite sure what you mean by "changing record types"?

In the standard configuration, the plugin only converts (into JSON) so-called "Data" records containing information about network flows measured by your exporter/probe. The structure description of these data records is given by the so-called (Options) Template records, which the probe sends to the collector as part of the NetFlow/IPFIX communication. These are basically just metadata.

To display them, just use the <templateInfo>true</templateInfo> option as shown in the help.

The frequency of occurrence of these records in the output of the plugin depends purely on the interval at which your probe sends them. As I mentioned before, templates are metadata within the NetFlow/IPFIX protocol. For example, when transferring IPFIX over TCP, the templates are typically sent at the beginning of the communication and then not at all. In the case of UDP, on the other hand, they come at an interval that is usually configurable on the probes. Look for "template refresh interval" or something similar.

Is this answer helpful? Let me know!

Lukas

Catwoolfii commented 5 months ago

@Lukas955, thanks for explanation, now it’s clear why these settings are needed. I'll better explain my goal: I need to remove unnecessary fields so as not to store unnecessary data in kafka. Is it possible to configure this in ipfixcol2?

Lukas955 commented 5 months ago

Unfortunately this feature is not supported at the moment. However, I have noted it among the features we might add in the future.

Lukas