Query Regarding Incoming and Outgoing Packet Rates in ipfixcol2 Collector

[Lidhoria]

We are encountering an issue while using the ipfixcol2 collector. Specifically, when sending traffic from a remote machine to the collector, we have noticed that the incoming rate of packets, received by the ipfixcol2 collector, is greater than the outgoing rate of packets originating from the remote machine. This situation seems counterintuitive and requires your expertise to understand and resolve.

Here is some information about our configuration:

We are running the ipfixcol2 collector using the command ./ipfixcol2 -vvv -c /opt/ipfixcol2/conf/startup.xml or ./ipfixcol2 -c /opt/ipfixcol2/conf/startup.xml.
Our collector setup involves the utilization of the UDP input plugin.
We have configured the collector to use a JSON-Kafka output plugin.
We observe that when we start the collector, there is a sudden and unexplained increase in the packet rate received by the collector, as we are sending outgoing packets from a different machine to the collector. The discrepancy between incoming and outgoing packet rates is unexpected.

We are reaching out to seek your suggestions on potential reasons for this discrepancy. We are eager to understand the root cause of this behavior and identify steps to rectify it. Could you kindly share your expertise on this matter?

[Lukas955]

Hi, I apologize for the delayed response.

I believe this might be due to the transfer of flow records to Kafka. Several reasons come to mind:

• The transfer to Kafka is via TCP, which generates a number of ACK packets that are sent back to the collector, increasing the number of packets received.
  • The librdkafka library buffers records and later sends them in larger batches i.e. one packet on receive does not automatically mean one packet on output. Additionally, output packets might be compressed if LZ4 compression is enabled. This reduces the number of output packets.
• The Kafka communication protocol itself may also generate some messages, but I can't see into this issue as this is in the domain of the librdkafka library.

Another situation could be, for example, if some NetFlow/IPFIX packets do not contain data records, but only contain templates describing the data. In this case, no data or packets will be generated by the Kafka module.

Lukas