PretendoNetwork / nex-viewer

Utility for parsing and (eventually) viewing NEX connections from WireShark network dumps
8 stars 5 forks source link

Replace PCAP(NG) parsers with custom ones #20

Closed jonbarrow closed 11 months ago

jonbarrow commented 11 months ago

We've been using two 3rd party modules for parsing pcap and pcapng captures. Both of these are very old and no longer maintained, and were prone to errors. I know we forked the pcapng parsing library, but honestly that codebase is frankly disgusting. This PR adds in two custom parsers for both capture types, improves the detection of which type is being used (sometimes @InternalLoss will give us dumps saved as a pcapng but they're really a pcap, so checking the file extension is not reliable), and it updates our dependencies to remove some vulnerable ones. It unifies the parsers into a similar API as well using generators so we can handle large files cleanly and without needing to rely on streams/events

Changes also include actually reading the IP header properly instead of blindly skipping chunks. The old system assumed that all packets were using an ethernet device, but now it expects the packet data to begin with the IP header (interface headers are parsed in the PCAPNG parser)

DaniElectra commented 11 months ago

Also, one of my PCAP network dumps doesn't show any packets when it should? No error is shown on the console

jonbarrow commented 11 months ago

Can you provide me the PCAP which is not working? I have tested PCAPNG captures as well as PCAP captures made both with HokakCTR and not and all mine have worked

jonbarrow commented 11 months ago

Reposting this from Discord:

The reason why that one pcap is not working is because it includes link layer headers in it's packets, unlike dumps provided by @InternalLoss made with tcpdump. Since the link layer has no magics, there's almost no good/simple way for us to handle these cases

If you open the pcap in WireShark and then export it again as a pcapng then WireShark "fixes" the capture, so I think that's the best move for now. Just try to use pcapng whenever possible

The only other way to support this with pcaps is either

Neither are great solutions imo

jonbarrow commented 11 months ago

I have also added in a small change which can automatically detect HokakuCTR dumps by checking for the header it now creates. This makes the manual option obsolete but I'm keeping it in anyway for now incase we find issues with the automatic detection during larger testing

jonbarrow commented 11 months ago

I see you approved the changes but I went ahead and fixed the timestamp issue. It was due to the old code just assuming the resolution was always the default libpcap resolution. Reading if_tsresol was indeed the fix

Screenshot from 2023-11-05 11-24-05

DaniElectra commented 11 months ago

Indeed, now it all works! (except for the pcap issue that you mentioned)

jonbarrow commented 11 months ago

I don't realistically think we're going to solve that pcap issue outside of converting non-working ones to pcapng's using WireShark. WireShark likely just does a TON of heuristics to parse everything which is way more complex than we really need to do tbh. Maybe if the servers weren't shutting down in a few months, so this tool would have possibly years more of widespread use, it would be justified but realistically this tools usefulness is also on a clock now, so I'd rather not spend an exceptional amount of time implementing small QOL features like that when we can just take the extra couple seconds to convert