search

CESNET / ipfixcol

IPFIXcol is an implementation of an IPFIX (RFC 7011) collector
Other
64 stars 37 forks source link

TCP forwarder and template handling #199

Open morph027 opened 6 years ago

morph027 commented 6 years ago

Hello, me again ;)

Found another little bugger in my test setup.

ipfixcol is forwarding to logstash via TCP. If there's too much load on logstash (TPC connection still alive), it somehow seems to drop/lost the templates or so (not sure) leading to messages not being processed (the whole logstash tcp receiving stack is bogus). Before ipfixcol as MITM it was working as the original sender we have is re-transmitting templates every x minutes even on TCP connections.

Does ipfixcol forward templates only when they are seen on a connection first? Or can we add something like the UDP resend setting for TCP also?

Lukas955 commented 6 years ago

Hi, I looked into the code and you are right that the forwarding plugin removes templates definitions that have been already sent. The purpose is to prevent template collisions by remapping Template IDs, if there are multiple flow sources (i.e. exporters connected to the collector) with the same Observation Domain ID (ODID).

However, all forwarded IPFIX Messages that consist of at least one Template definition (that hasn't been sent yet) are marked as "mandatory delivery". In other words, if a destination is busy and the plugin is not able to pass the message into a TCP socket (or store it into a temporary buffer for next time delivery), connection is closed to prevent sending malformed/unreadable messages. Moreover, the plugin sends only Data Sets (i.e. flow records) with known definitions of templates. As a result, only fully interpretable messages should be send to the destination.

Note: Closed connections are later reconnected and all known template definitions are send at the beginning of communication.

morph027 commented 6 years ago

Hi.

Thanks for the explanation. Makes sense ;)

The thing is that the logstash TCP socket never is busy somehow (no log entries from ipfixol about busy/broken connection), it just starts loosing it's templates and then complains. I totally blame logstash for this ;)

Lukas955 commented 6 years ago

It's a small hack, but if you want to try retransmission of the templates over TCP, you can just remove this condition (not tested) and recompile the collector: https://github.com/CESNET/ipfixcol/blob/74cdcd29893b4d8acfdbd916dfd0005e8a560f17/base/src/storage/forwarding/destination.c#L528

Retransmission interval is defined by the configuration parameter <udpTemplateRefreshTimeout> (by default 300 seconds)

thorgrin commented 6 years ago

What are you trying to achieve? I've used logstash in the past, but it is not very fast. The IPFIXcol's json plugin has an experimental feature which allows to send messages directly to Kafka, if that helps.