Unpack requires a buffer of 14 bytes

[Willtl]

I'm trying to perform some preprocessing on some .pcap files to perform anomaly detection using neural networks.

This is the dataset: https://github.com/yungshenglu/USTC-TFC2016

I managed to open and work with the benign data normally (https://github.com/yungshenglu/USTC-TFC2016/tree/master/Benign).

However, when I try to open any .pcap inside the malware folder (https://github.com/yungshenglu/USTC-TFC2016/tree/master/Malware), I get the following error when I call eth = dpkt.ethernet.Ethernet(frame)

Traceback (most recent call last):
  File "C:\Users\wlb\miniconda3\envs\pytorchcuda\lib\site-packages\dpkt\dpkt.py", line 87, in __init__
    self.unpack(args[0])
  File "C:\Users\wlb\miniconda3\envs\pytorchcuda\lib\site-packages\dpkt\ethernet.py", line 129, in unpack
    dpkt.Packet.unpack(self, buf)
  File "C:\Users\wlb\miniconda3\envs\pytorchcuda\lib\site-packages\dpkt\dpkt.py", line 177, in unpack
    struct.unpack(self.__hdr_fmt__, buf[:self.__hdr_len__])):
struct.error: unpack requires a buffer of 14 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/wlb/Documents/GitHub/dl-network-anomaly-detection/process_pcap.py", line 196, in <module>
    main()
  File "C:/Users/wlb/Documents/GitHub/dl-network-anomaly-detection/process_pcap.py", line 190, in main
    process_packets(args)
  File "C:/Users/wlb/Documents/GitHub/dl-network-anomaly-detection/process_pcap.py", line 121, in process_packets
    eth = dpkt.ethernet.Ethernet(frame)
  File "C:\Users\wlb\miniconda3\envs\pytorchcuda\lib\site-packages\dpkt\ethernet.py", line 76, in __init__
    dpkt.Packet.__init__(self, *args, **kwargs)
  File "C:\Users\wlb\miniconda3\envs\pytorchcuda\lib\site-packages\dpkt\dpkt.py", line 90, in __init__
    raise NeedData('got %d, %d needed at least' % (len(args[0]), self.__hdr_len__))
dpkt.dpkt.NeedData: got 4, 14 needed at least

I tried to open the same malware .pcap files on Wireshark and it loads it correctly.

[obormot]

I tried opening https://github.com/yungshenglu/USTC-TFC2016/blob/master/Malware/Miuref.pcap in Wireshark and it shows the 1st packet is truncated. You're likely hitting the same error in dpkt, saying it needs more data to process the packet. This is the expected behavior. You can modify your code to catch the NeedData exception and move on to process the next packet. Please reopen the issue if you believe there's a bug in dpkt.

[brifordwylie]

@Willtl also just for an example of handling truncated data you can take a peek at this example: https://github.com/kbandla/dpkt/blob/master/examples/print_dns_truncated.py