Extremely slow processing of pcap files

[nuschpl]

I used to process ~15GB pcap files with https://github.com/lgandx/PCredz and I believed it's slow and not optimal tool. But BruteShark comparing to it is extremely slow. Filtering 15GB pcap with wireshark for NTLMSSP packets takes less than a minute on my 16core, 32GB hardware, it results in 3MB file which is processed by BruteShark almost the same ammount of time - something is not optimal here. I've found it utilizes only 1 CPU core and use limited amount of RAM for processing, is there any prefered way to implement threads and caching here ? I can contribute

[odedshimon]

Hi @nuschpl !

Thanks for letting me know.

If you used BruteSharkCli it is It makes sense that the run time was long since it is reconstructing all the TCP / UDP sessions by default.

Unfortunately the user-interface is currently does not provide the required capabilities for use a configuration that is appropriate for a particular analysis.. (let's say in your case it is enough to run only the hashes module without building sessions).

BruteSharkDesktop although provide this capabilities (see picture attached), you may prefer to use it for the pcap you described.

In the near future we plan to make significant refactor that will also allow BruteSharkCli to run as a single command line with all the options that the tool provides (this will also solve the other issue that you have opened).

We are welcome contributors, feel free to suggest a design for BruteSharkCli or even implement it. Also feel free to contribute to any other part of BruteShark.

[nuschpl]

Hi, I'm not sure sacrificing some functionality (reconstructing sessions in this case) is the only solution if current implementation is slow. I have at least two suggestions:

1. Use all CPU cores- in my case one core was 100% , while other 15 cores/threads were idling - it was not I/O issue.
2. Use moving window for recreating the session - sorry for ignorance - didn't looked into the code yet, but looking at time of processing at very end - 10TCP sessions per second it seems you do some comparisons across entrie PCAP file so processing time increases with every loaded packet. But TCP sessions usually don't last so long - at maximum few minutes when no related packet is transfered - so packes could be loaded into caching queue with:
   • one thread could append loaded packets from multiple PCAPs to the end of queue and up to user defined RAM usage
   • user defined (defaulting to avaiable CPU cores -2 ) amount of threads could process the data with current approach
   • separate thread could remove the oldest packets from caching queue marked as processed and not used since limit (e.g 6 minutes(in relation to PCAP timestamps) and removing those already marked

[odedshimon]

It's definitely not the only solution, but a trivial one (and useful for the general user experience as well). Thank you for suggesting this, I will consider it as possible improvements. Also, feel free to fork this project and implement any of this solutions, as well as pull-requesting to this repo in case you will decide to do it.

[aviadtamir]

@nuschpl

there's a branch in this repo in which i'm trying to overcome this issue, that a look if you want to :)

a little glimpse to what i've already accomplished;

https://user-images.githubusercontent.com/24754559/102596487-e82e3280-4121-11eb-8705-6871f6e39d2d.mp4

[odedshimon]

BruteSharkCli now has two modes: single command and shell mode. The single command mode works by getting all the relevant parameters for the processing and then printing the results to stdout or files. The shell mode allows to perform each step individually.

You can just set the credentials module and configure an output directory and all your hashes will be exported automatically.

Example Video: https://youtu.be/GjGV5vwIGi8

Please see the 'Usage' section for detailed documentation and examples.