IPFIX "Illegal message lengthX" Error

[tal-zvon]

ilmarh's example from: https://github.com/britram/python-ipfix/issues/17 works great for Netflow v9 (at least with my limited testing).

When I add import ipfix.reader, and change his r = ipfix.v9pdu.from_stream(self.rfile) to say r = ipfix.reader.from_stream(self.rfile) instead, it always throws an exception for me. The traceback says:

...
for rec in r.namedict_iterator():
...
ipfix.template.IpfixDecodeError: ('Illegal message length2',)

The 2 isn't always a 2.

The flow is coming from a Mikrotik device. Am I doing something wrong? Or is the Mikrotik sending some improperly-formatted packets? PRTG seemed to have no issues collecting IPFIX data from the Mikrotik.

My full code looks like this:

#!/usr/bin/env python3

import socketserver
#import ipfix.v9pdu
import ipfix.reader
import ipfix.ie

import argparse

ap = argparse.ArgumentParser(description="Dump IPFIX files collected over UDP")
ap.add_argument('--spec', metavar='specfile', help='iespec file to read')
args = ap.parse_args()

ipfix.ie.use_iana_default()
ipfix.ie.use_5103_default()

if args.spec:
    ipfix.ie.use_specfile(args.spec)

msgcount = 0
templates = {}
accepted_tids = set()

def dummy1(o,b):
    pass

class CollectorDictHandler(socketserver.DatagramRequestHandler):
    def handle(self):
        global msgcount
        msgcount = msgcount + 1
        reccount = 0

        print("connection from " + str(self.client_address))
        r = ipfix.reader.from_stream(self.rfile)

        try:
            r.templates = templates[str(self.client_address)]
        except KeyError as e:
            templates[str(self.client_address)] = {}
            r.templates = templates[str(self.client_address)]

        r.accepted_tids = accepted_tids
        r.unknown_data_set_hook = dummy1

        for rec in r.namedict_iterator():
            print("--- record %u in message %u from %s---" %
                 (reccount, msgcount, str(self.client_address)))
            reccount += 1
            for key in rec:
                 print("  %30s => %s" % (key, str(rec[key])))

        print("reccount = "+str(reccount))

ss = socketserver.UDPServer(("", 1234), CollectorDictHandler)
ss.serve_forever()

[tal-zvon]

Sounds like it may be related to this: https://github.com/logstash-plugins/logstash-codec-netflow/issues/58#issuecomment-274417666. Can anyone confirm?

[tal-zvon]

Packet captures confirm that IPFIX from Mikrotiks had lengths < 20, which makes python-ipfix raise an IpfixDecodeError exception at around line 323 of message.py.

Mikrotik's RouterOS 6.37.5 had the bug, but updating to the next version up (6.38) fixed the issue, so if anyone else finds this experiencing the same problem with a Mikrotik, you're either stuck using Netflow 9, which seems to work fine, or you need to update to at least RouterOS 6.38.