Pyrit crashes when importing large password file to MySQL database.

[GoogleCodeExporter]

Hi there!

I'm attempting to import a large (1.3GB) text file full of passwords into 
Pyrit's MySQL database and it has failed repeatedly.

The file in question is very high-quality (UNIQPASS v11, 
https://dazzlepod.com/uniqpass/) so I'm confident that it's not causing the 
issue.  I'm using a MySQL database as my default storage as configured with 
~/.pyrit/config.

My system is a quad-core AMD CPU running 64-bit Ubuntu 12.10.

When I run <pyrit -i UNIQPASS.txt import_passwords> it runs for some variable 
length of time before crashing.  (Same issue is present in a detached screen 
session and in a remote shell.)

Strange thing is, it produces a different error and crashes after a different 
number of lines each time I run it (stars are mine for readability):

****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (567195.0 lines/s)... Segmentation fault (core dumped)
*****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
97100000 lines read (114239.9 lines/s)... Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 115, in initFromArgv
    func(self, **options)
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 355, in import_passwords   
    pwstore.store_password(line)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 363, in store_password
    self._flush_bucket(pw_h1, pw_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 1196, in _flush_bucket
    bucket.difference_update(db_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 926, in __iter__
    self.collection.unpack(self.collection_buffer)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 287, in unpack
    raise DigestError("Digest check failed")
cpyrit.storage.DigestError: Digest check failed
****************************************************************************
Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
45000000 lines read (459279.8 lines/s)... Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 115, in initFromArgv
    func(self, **options)
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 355, in import_passwords   
    pwstore.store_password(line)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 363, in store_password
    self._flush_bucket(pw_h1, pw_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 1199, in _flush_bucket
    session.add(PAW2_DBObject(pw_h1, bucket))
  File "<string>", line 4, in __init__
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 98, in initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 917, in __init__
    key, collection_buffer = PAW2_Buffer(self.collection).pack()
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 276, in pack
    b = zlib.compress('\n'.join(self.collection), 1)
TypeError: sequence item 9425: expected string,  found
****************************************************************************

Considering that Pyrit is supposed to be a robust and scalable tool that can 
manage massive databases, I hope this is just me! Please let me know if there 
is anything I can do to help troubleshoot or if I'm missing something.

Thanks for making a great tool.
Alex

Original issue reported on code.google.com by iAlex.R...@gmail.com on 13 Feb 2013 at 5:00

[GoogleCodeExporter]

I've continued to run it and continued to get failures.  I have the core dumps 
for these and attached the dump for the second error in this list.

****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  Segmentation fault 
(core dumped)
****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (566317.4 lines/s)... Segmentation fault (core dumped)
****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (575655.4 lines/s)... Segmentation fault (core dumped)
****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
26800000 lines read (408332.9 lines/s)... Segmentation fault (core dumped)
****************************************************************************
^This last one was importing a different wordlist than the others, note the 
different number of lines after which it segfaulted.

Original comment by iAlex.R...@gmail.com on 14 Feb 2013 at 7:33

Attachments:

• core2.bz2

[GoogleCodeExporter]

I should also mention that I can successfully import the file if I break it up 
into small subunits, but the failure only occurs when dealing with a very large 
file.

Original comment by iAlex.R...@gmail.com on 19 Feb 2013 at 6:15

[GoogleCodeExporter]

Alright well seeing that this project a ghost town nowadays, I've put together 
a hacky sort of way to solve the problem by myself.  This script basically 
splits a large passwordlist file into a bunch of smaller files and imports them 
individually, waiting 15 seconds between each one.  This seems to mitigate the 
problem.

#!/bin/bash

PREFIX="temporarysplitfile-"
NUMBEROFSPLITS=200

echo "Which file would you like to import?"
read FILENAME
echo "Using file $FILENAME"

echo "Splitting file $FILENAME into $NUMBEROFSPLITS parts with prefix $PREFIX."
split -n $NUMBEROFSPLITS $FILENAME $PREFIX
echo "File successfully split."

for i in $PREFIX*
do
    echo "Importing file $i"
    pyrit -i $i import_passwords
    echo "Waiting 15 seconds"
    sleep 15
done

echo "Import complete!"
rm $PREFIX*
echo "Split files removed."

Original comment by iAlex.R...@gmail.com on 19 Feb 2013 at 9:10

[GoogleCodeExporter]

Thanks for sharing script !

Original comment by ccchurch...@gmail.com on 12 Mar 2013 at 9:57

[GoogleCodeExporter]

Great script thanks

Original comment by mar...@hempstock.it on 30 Jul 2013 at 10:41

[GoogleCodeExporter]

have the same issue. All import passwords die with the above mentioned error 
once the db file reaches 10KB (exact line changes, obviously, but the output 
file size is always 10KB when crashed).

Running 0.4.1-dev (svn r308) 

Original comment by wat...@gmail.com on 25 Jun 2014 at 5:15