search

hmaverickadams / breach-parse

A tool for parsing breached passwords
1.8k stars 523 forks source link

Provide compressed version of BreachCompilation #4

Open CosmicHorrorDev opened 4 years ago

CosmicHorrorDev commented 4 years ago

Hello, thanks for the great project!

I was wondering if you would be open to providing a compressed version of BreachCompilation for people like me who have very slow internet. It is nice to provide the magnet link for the uncompressed version so people have the option to pick and choose if they only want specific files, but on the other hand it took 2 full days for my download to finish. Using zstd -r --ultra --long -c -20 -T0 > breach.zst (-T0 should be multithreaded, but only used a single thread for me likely because of the high compression level) got it down from 41.1GiB to 10.5 GiB, and just doing the default compression level which was much faster (zstd -r -c -T0 > breach.zst) got it down to 13GiB. Either way, this is a much smaller size that would take far less time for me to download and decompressing the 10.5 GiB file took me less than 4 minutes (On an SSD with a relatively old CPU Intel i5-5200U @ 2.7GHz). Providing both options leaves the choice up to the user, but I think a lot of people likely want all the files so most people would probably choose a faster download over using a bit of extra drive space until everything is decompressed.

CosmicHorrorDev commented 4 years ago

I guess it would be good to note that you should use something like tar to handle compressing and extracting directories since zstd won't. Something like tar --zstd -cf breach.tar.zst BreachCompilation and tar --zstd -xf breach.tar.zst or for older versions of tar tar --use-compress-program zstd -cf breach.tar.zst BreachCompilation and tar --use-compress-program zstd -xf breach.tar.zst for older versions of tar. I picked zstd for the fast extraction speed, but any decent compression format would be nice.

ghost commented 3 years ago

How can i download the magnet link ??

I have tried to download from online, its showing to upgrade the user to get the file which is huge 1GB+.

Thanks in advance.

philipvantuyl commented 2 years ago

I used Deluge to download the file. You can get it by typing sudo apt-get install deluge into your Kali command line (remove the sudo if you're running as root). Then just copy and paste the address into the URL box in Deluge. It will get the list of files. If you want to put it in the directory that Breach-Parse expects it to be in, put it in opt/breach-parse/BreachCompilation/data