search

bitkeks / python-netflow-v9-softflowd

PyPI "netflow" package. NetFlow v9 parser, collector and analyzer implemented in Python 3. Developed and tested with softflowd
https://bitkeks.eu/blog/2016/08/collecting-netflow-v9-on-openwrt.html
MIT License
110 stars 56 forks source link

Error while handling packets with no template #4

Closed skoroneos closed 6 years ago

skoroneos commented 6 years ago

Greetings. I used a mikrotik generated netflow and found there is a problem when the full template is not send with every packet.

i.e

Listening on interface :2055
Starting the NetFlow listener
Received data from 192.168.255.254, length 204
Processed ExportPacket with 2 flows.
Received data from 192.168.255.254, length 1268
Exception happened during processing of request from ('192.168.255.254', 2055)
Traceback (most recent call last):
  File "/usr/lib/python3.4/socketserver.py", line 305, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python3.4/socketserver.py", line 331, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.4/socketserver.py", line 673, in __init__
    self.handle()
  File "main.py", line 72, in handle
    export = ExportPacket(data, self.TEMPLATES)
  File "/home/stelios/projects/netflow/python-netflow-v9-softflowd/src/netflow/collector_v9.py", line 266, in __init__
    dfs = DataFlowSet(data[offset:], self.templates)
  File "/home/stelios/projects/netflow/python-netflow-v9-softflowd/src/netflow/collector_v9.py", line 146, in __init__
    fkey = field_types[field.field_type]
KeyError: 225
Received data from 192.168.255.254, length 204
Processed ExportPacket with 2 flows.

The first packet contains a complete template flow for two templates (with id;s 256 and 257)
256 is for ipv4 traffic and id 257 is for IPV6. Since the IPV6 has different fields when a data packet arrives for IPV4 it does not have the fields.

According to the specs, the sender does not have to send the entire template everytime and the recipient has to cache the template and use it, till it expires or its resend. softflowd sends only one template with id 1024 so probably this is why you haven't seen it.

i.e

https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html

bitkeks commented 6 years ago

Hello, thanks for your bug report! I see two things:

  1. The persistence of templates is an issue, yes. I thought about solving it by using a unique identifier for each template and save it to disk, so it could be reloaded later if the collector is restarted and loses state. The current implementation should be abel to handle different templates in the same flow, saved in self.TEMPLATES.

  2. Your tracelog indicates a KeyError with key 225 because the template seems to use a field identifier which is not found in the hardcoded list of field types. I'd like to debug this further, but my guess is that your NetFlow sender uses custom fields which are not defined in the specs.

The best solution would be to catch this error and produce debugging output. Maybe you find the time to narrow down the issue, it would improve the lib greatly to catch another edge-case :wrench:

skoroneos commented 6 years ago

I think you are right on 2. There are 4 field types 225,226,227 and 228 which are not in the list. These, as i found out, are not custom fields but an extension to the V9 spec by Cisco to handle the ASA
https://www.cisco.com/c/en/us/td/docs/security/asa/special/netflow/guide/asa_netflow.html It also defined a few other types namely in the XXXXX range which are custom defined by manufacturers PaloAlto networks defines 346,56701,56702

https://www.paloaltonetworks.com/documentation/80/pan-os/pan-os/monitoring/netflow-monitoring/netflow-templates I can assume others define also their own (Don't you love standards :D)

For 1. I have not looked into how the code handles the templates

skoroneos commented 6 years ago

This is the error when the template is now known (i.e collector (re)started in the middle of the stream) In my case the router will resend the templates after 20 packets and it syncs without an issue

Listening on interface :2055
Starting the NetFlow listener
Received data from 192.168.255.254, length 508
Exception happened during processing of request from ('192.168.255.254', 2055)
Traceback (most recent call last):
  File "/usr/lib/python3.4/socketserver.py", line 305, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python3.4/socketserver.py", line 331, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.4/socketserver.py", line 673, in __init__
    self.handle()
  File "main.py", line 72, in handle
    export = ExportPacket(data, self.TEMPLATES)
  File "/home/stelios/projects/netflow/python-netflow-v9-softflowd/src/netflow/collector_v9.py", line 271, in __init__
    dfs = DataFlowSet(data[offset:], self.templates)
  File "/home/stelios/projects/netflow/python-netflow-v9-softflowd/src/netflow/collector_v9.py", line 141, in __init__
    template = templates[self.template_id]
KeyError: 256
bitkeks commented 6 years ago

are not custom fields but an extension to the V9 spec by Cisco to handle the ASA

It also defined a few other types namely in the XXXXX range

Thanks for the research! It won't be a problem to add these IDs to the list, since the field_type struct can hold up to 65535 identifiers. I'll add them in the next days if possible! (also adding a try-except block…)

This is the error when the template is now known (i.e collector (re)started in the middle of the stream)

Did you mean is not known?

skoroneos commented 6 years ago

Templates periodically expire if they are not refreshed. Templates can be refreshed in two ways. A template can be resent every N number of export packets. A template can also be sent on a timer, so that it is refreshed every N number of minutes. Both options are user configurable.

If a collector (i.e python netflow) connects to the stream after the template is send it has no way of knowing what the flow data in the packets mean, until the template is resend. From my little experimentation some devices collect and aggregate data for longer periods of time (i.e softflowd has a default of 5 minutes) and others like mikrotik send packets every few seconds. Those that aggregate data for longer periods (usually) send the templates also before the data, while the ones that send more frequent packets send several data flow packets before the template is resend, probabably to save on bandwidth.

I guess the best way to handle this to have a message that says something similar to "Data received but no know template exists" and continue operations waiting for the template

bitkeks commented 6 years ago

I have added the referenced additional field types, if you'd like to test it out!

skoroneos commented 6 years ago

Tested it and works fine. You can close the ticket