Reading malformed (network) binaries

[H4rryK4ne]

Hi,

I am not sure, if this is the right place for this issue, but since we have this issue with the generated python code I open the issue here.

We have a pcap file, which contains ethernet frames, but some of them are malformed, e.g. the pcap package header says its an ethernet frame, but the frame has only one byte. If I try to parse the complete pcap-file into a pcap object the parser fails due to reading less than expected bytes and the an exception is raised.

Is there a possibility to skip malformed packages?

Currently, we removed the link to the ethernet frames in the pcap.ksy and parse each package in a try catch block.

[GreyCat]

Not a Python-runtime-specific question, so moving to central repo.

For reading something continuous such as pcap files, I'd advise to just extract the pcap parsing loop into your code and implement exception handling logic there. For example, this is the pcap parsing loop generated by KS:

        self.hdr = self._root.Header(self._io, self, self._root)
        self.packets = []
        i = 0
        while not self._io.is_eof():
            self.packets.append(self._root.Packet(self._io, self, self._root))
            i += 1

You can just invoke Header / Packet parsing manually, handling exceptions as necessary by your project, i.e. something like:

f = open('/path/to/your.pcap', 'rb')

root = Pcap.__new__(Pcap)
root._io = KaitaiStream(f)
root.hdr = Pcap.Header(root._io, root, root)
while not root._io.is_eof():
    try:
        pkt = Pcap.Packet(root._io, root, root)

        # Do something with each packet
        print(pkt.ts_sec)
    except:
        print("Parsing exception occurred, skipping packet")

[KOLANICH]

1. I agree with @GreyCat. I have used this approach in https://gitlab.com/KOLANICH/USBPcapOdinDumper . BTW, though it is for usb captures, it has architecture allowing to easily modify it to other stacked protocols.

2. If you want an automated build, you can create a patch file between the output of KSC and the modified file. This works pretty fine. And is extremily useful for debugging and prototyping.

3. I have a semi-finished draft of pcapng I currently don't have any time to improve in kaitai_struct_formats repo in my account, you also may find it useful.

[H4rryK4ne]

Great and many thanks!

Best regards!