unpack requires a buffer of 4 bytes

oortega commented 5 years ago

Hello I used the collector with 2 mikrotik routers. The router1 has little traffic, about 150 connections and it worked very well,

but the router2 has enough traffic about 11200 connections, here I had the following exception

Exception happened during processing of request from ('192.168.19.250', 9000)
Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.6/socketserver.py", line 721, in __init__
    self.handle()
  File "main.py", line 74, in handle
    export = ExportPacket(data, self.templates)
  File "/home/kukulcan/Documentos/github/python-netflow-v9-softflowd/src/netflow/collector_v9.py", line 316, in __init__
    tfs = TemplateFlowSet(data[offset:])
  File "/home/kukulcan/Documentos/github/python-netflow-v9-softflowd/src/netflow/collector_v9.py", line 270, in __init__
    field_type, field_length = struct.unpack('!HH', data[offset:offset+4])
struct.error: unpack requires a buffer of 4 bytes

Any idea how to increase the buffer?

Thanks.

bitkeks commented 5 years ago

Hello, thanks for the report! So, my guess is that the data[offset:offset+4] slice does not actually hold 4 bytes, but less, indicating that the data payload is not long enough. Reproducible through the following test cases:

# 4 bytes
>>> data = b"\x00\x01\x00\x01"
>>> struct.unpack("!HH", data)
(1, 1)

# Too many bytes
>>> data = b"\x00\x01\x00\x01\x00"
>>> struct.unpack("!HH", data)
struct.error: unpack requires a buffer of 4 bytes

# 4 bytes slice
>>> struct.unpack("!HH", data[0:4])
(1, 1)

# too short to fill slice
>>> data = b"\x00\x00\x00"
>>> struct.unpack("!HH", data[:4])
struct.error: unpack requires a buffer of 4 bytes

In this case, we'd have to investigate, why. Looking into the NetFlow TemplateFlowSet specs, the length of the fields which are iterated over are fixed at 32 bits/4 bytes.

My guess is therefore, that something is broken in transit and the packets are corrupted. If you have some sample data, we might have a chance to find the problem. This could for example be a second packet capturing file, which captures the received packets at the time the error appears.

You can optionally anonymize and shorten any uploads, since everyone with access to this issue has access to attachments.. :wink:

oortega commented 5 years ago

I was a fool, I was using version 5 of netflow instead of v9.

bitkeks / python-netflow-v9-softflowd

unpack requires a buffer of 4 bytes #7