Closed jonbarrow closed 11 months ago
Also, one of my PCAP network dumps doesn't show any packets when it should? No error is shown on the console
Can you provide me the PCAP which is not working? I have tested PCAPNG captures as well as PCAP captures made both with HokakCTR and not and all mine have worked
Reposting this from Discord:
The reason why that one pcap is not working is because it includes link layer headers in it's packets, unlike dumps provided by @InternalLoss made with tcpdump. Since the link layer has no magics, there's almost no good/simple way for us to handle these cases
If you open the pcap in WireShark and then export it again as a pcapng then WireShark "fixes" the capture, so I think that's the best move for now. Just try to use pcapng whenever possible
The only other way to support this with pcaps is either
Neither are great solutions imo
I have also added in a small change which can automatically detect HokakuCTR dumps by checking for the header it now creates. This makes the manual option obsolete but I'm keeping it in anyway for now incase we find issues with the automatic detection during larger testing
I see you approved the changes but I went ahead and fixed the timestamp issue. It was due to the old code just assuming the resolution was always the default libpcap resolution. Reading if_tsresol
was indeed the fix
Indeed, now it all works! (except for the pcap issue that you mentioned)
I don't realistically think we're going to solve that pcap issue outside of converting non-working ones to pcapng's using WireShark. WireShark likely just does a TON of heuristics to parse everything which is way more complex than we really need to do tbh. Maybe if the servers weren't shutting down in a few months, so this tool would have possibly years more of widespread use, it would be justified but realistically this tools usefulness is also on a clock now, so I'd rather not spend an exceptional amount of time implementing small QOL features like that when we can just take the extra couple seconds to convert
We've been using two 3rd party modules for parsing pcap and pcapng captures. Both of these are very old and no longer maintained, and were prone to errors. I know we forked the pcapng parsing library, but honestly that codebase is frankly disgusting. This PR adds in two custom parsers for both capture types, improves the detection of which type is being used (sometimes @InternalLoss will give us dumps saved as a pcapng but they're really a pcap, so checking the file extension is not reliable), and it updates our dependencies to remove some vulnerable ones. It unifies the parsers into a similar API as well using generators so we can handle large files cleanly and without needing to rely on streams/events
Changes also include actually reading the IP header properly instead of blindly skipping chunks. The old system assumed that all packets were using an ethernet device, but now it expects the packet data to begin with the IP header (interface headers are parsed in the PCAPNG parser)