Pyrit crashes when importing large password file to MySQL database.

[GoogleCodeExporter]

Hi there!

I'm attempting to import a large (1.3GB) text file full of passwords into 
Pyrit's MySQL database and it has failed repeatedly.

The file in question is very high-quality (UNIQPASS v11, 
https://dazzlepod.com/uniqpass/) so I'm confident that it's not causing the 
issue.  I'm using a MySQL database as my default storage as configured with 
~/.pyrit/config.

My system is a quad-core AMD CPU running 64-bit Ubuntu 12.10.

When I run <pyrit -i UNIQPASS.txt import_passwords> it runs for some variable 
length of time before crashing.  (Same issue is present in a detached screen 
session and in a remote shell.)

Strange thing is, it produces a different error and crashes after a different 
number of lines each time I run it (stars are mine for readability):

****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (567195.0 lines/s)... Segmentation fault (core dumped)
*****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
97100000 lines read (114239.9 lines/s)... Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 115, in initFromArgv
    func(self, **options)
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 355, in import_passwords   
    pwstore.store_password(line)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 363, in store_password
    self._flush_bucket(pw_h1, pw_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 1196, in _flush_bucket
    bucket.difference_update(db_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 926, in __iter__
    self.collection.unpack(self.collection_buffer)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 287, in unpack
    raise DigestError("Digest check failed")
cpyrit.storage.DigestError: Digest check failed
****************************************************************************
Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
45000000 lines read (459279.8 lines/s)... Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 115, in initFromArgv
    func(self, **options)
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 355, in import_passwords   
    pwstore.store_password(line)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 363, in store_password
    self._flush_bucket(pw_h1, pw_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 1199, in _flush_bucket
    session.add(PAW2_DBObject(pw_h1, bucket))
  File "<string>", line 4, in __init__
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 98, in initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 917, in __init__
    key, collection_buffer = PAW2_Buffer(self.collection).pack()
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 276, in pack
    b = zlib.compress('\n'.join(self.collection), 1)
TypeError: sequence item 9425: expected string,  found
****************************************************************************

Considering that Pyrit is supposed to be a robust and scalable tool that can 
manage massive databases, I hope this is just me! Please let me know if there 
is anything I can do to help troubleshoot or if I'm missing something.

Thanks for making a great tool.
Alex

Original issue reported on code.google.com by iAlex.R...@gmail.com on 13 Feb 2013 at 5:00

[GoogleCodeExporter]

I've continued to run it and continued to get failures.  I have the core dumps 
for these and attached the dump for the second error in this list.

****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  Segmentation fault 
(core dumped)
****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (566317.4 lines/s)... Segmentation fault (core dumped)
****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (575655.4 lines/s)... Segmentation fault (core dumped)
****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
26800000 lines read (408332.9 lines/s)... Segmentation fault (core dumped)
****************************************************************************
^This last one was importing a different wordlist than the others, note the 
different number of lines after which it segfaulted.

Original comment by iAlex.R...@gmail.com on 14 Feb 2013 at 7:33

Attachments:

• core2.bz2

[GoogleCodeExporter]

I should also mention that I can successfully import the file if I break it up 
into small subunits, but the failure only occurs when dealing with a very large 
file.

Original comment by iAlex.R...@gmail.com on 19 Feb 2013 at 6:15

[GoogleCodeExporter]

Alright well seeing that this project a ghost town nowadays, I've put together 
a hacky sort of way to solve the problem by myself.  This script basically 
splits a large passwordlist file into a bunch of smaller files and imports them 
individually, waiting 15 seconds between each one.  This seems to mitigate the 
problem.

#!/bin/bash

PREFIX="temporarysplitfile-"
NUMBEROFSPLITS=200

echo "Which file would you like to import?"
read FILENAME
echo "Using file $FILENAME"

echo "Splitting file $FILENAME into $NUMBEROFSPLITS parts with prefix $PREFIX."
split -n $NUMBEROFSPLITS $FILENAME $PREFIX
echo "File successfully split."

for i in $PREFIX*
do
    echo "Importing file $i"
    pyrit -i $i import_passwords
    echo "Waiting 15 seconds"
    sleep 15
done

echo "Import complete!"
rm $PREFIX*
echo "Split files removed."

Original comment by iAlex.R...@gmail.com on 19 Feb 2013 at 9:10

[GoogleCodeExporter]

Thanks for sharing script !

Original comment by ccchurch...@gmail.com on 12 Mar 2013 at 9:57

[GoogleCodeExporter]

Great script thanks

Original comment by mar...@hempstock.it on 30 Jul 2013 at 10:41

[GoogleCodeExporter]

have the same issue. All import passwords die with the above mentioned error 
once the db file reaches 10KB (exact line changes, obviously, but the output 
file size is always 10KB when crashed).

Running 0.4.1-dev (svn r308) 

Original comment by wat...@gmail.com on 25 Jun 2014 at 5:15

[JPaulMora]

Please confirm if this is still a problem.

[dTechLife]

This is still a problem in the latest 0.5.1 release. I have no problem beta testing for you if you still develop actively for this project.

[JPaulMora]

Hello, thanks for your feedback. I've downloaded the coredumps above and will start tracing the error. Any updates please feel free to post them here, if you have different stack traces/errors that may help too.

[dTechLife]

I have also gotten an error using the script provided by GoogleCodeExporter, stack dump follows. Pyrit 0.5.1 (C) 2008-2011 Lukas Lueg - 2015 John Mora https://github.com/JPaulMora/Pyrit This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/mydb'... connected. 38000000 lines read (4515.9 lines/s)... Traceback (most recent call last): File "/usr/local/bin/pyrit", line 6, in pyrit_cli.Pyrit_CLI().initFromArgv() File "/usr/local/lib/python2.7/dist-packages/pyrit_cli.py", line 118, in initFromArgv func(self, options) File "/usr/local/lib/python2.7/dist-packages/pyrit_cli.py", line 375, in import_passwords pwstore.store_password(line) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 319, in store_password self._flush_bucket(pw_h1, pw_bucket) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 1154, in _flush_bucket bucket.difference_update(db_bucket) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 882, in iter return iter(self.collection) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 876, in getattr self.collection = PAW2_Buffer(self.collection_buffer) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 231, in init** raise DigestError("Digest check failed") cpyrit.storage.DigestError: Digest check failed

[dTechLife]

Also, any clue why importing slows down around "19100000 lines read (0.0 lines/s)... )... " and 28000000 lines also lags.

[dTechLife]

Here is another one for you, sorry :P It was completed importing and failed during the flush. Connecting to storage at 'mysql://localhost/mydb'... connected. 50000000 lines read. Flushing buffers.... Traceback (most recent call last): File "/usr/local/bin/pyrit", line 6, in pyrit_cli.Pyrit_CLI().initFromArgv() File "/usr/local/lib/python2.7/dist-packages/pyrit_cli.py", line 118, in initFromArgv func(self, options) File "/usr/local/lib/python2.7/dist-packages/pyrit_cli.py", line 380, in import_passwords self.tell("\r%i lines read. Flushing buffers..." % (i + 1)) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 287, in exit self.flush_buffer() File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 297, in flush_buffer self._flush_bucket(pw_h1, pw_bucket) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 1154, in _flush_bucket bucket.difference_update(db_bucket) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 882, in iter return iter(self.collection) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 876, in getattr self.collection = PAW2_Buffer(self.collection_buffer) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 231, in init** raise DigestError("Digest check failed") cpyrit.storage.DigestError: Digest check failed

[JPaulMora]

thanks! will check on that and keep you posted.

[dTechLife]

Send me revisions please, I would love to help out, ill keep running the same 200 files through import using the script up there by googlecodeexporter, modified. ^ I will post my upgraded import wordlist script for people to use until the update rolls out. Is there a dev version of pyrit I can swap to?

[dTechLife]

continue.sh

#!/bin/bash 

PREFIX="splitfile-"

for i in $PREFIX*
do
    echo "Importing file $i"
    pyrit -i $i import_passwords
    if [ $? == "0" ]; then
        rm $i
    else
        pyrit -i $i import_passwords
        if [ $? == "0" ]; then
            rm $i
        else
            echo "File $i seems to be giving Pyrit some issues. Skipping it for now, not removing. Run again later with this file alone.."
        fi
    fi
done

echo "Import complete!"
echo "Split files removed if imported."

split.sh

#!/bin/bash

PREFIX="splitfile-"
NUMBEROFSPLITS=200

echo "Which file would you like to import?"
read FILENAME
echo "Using file $FILENAME"

echo "Splitting file $FILENAME into $NUMBEROFSPLITS parts with prefix $PREFIX."
split -n $NUMBEROFSPLITS $FILENAME $PREFIX
echo "File successfully split."
echo "Removing $FILENAME."
rm $FILENAME

for i in $PREFIX*
do
    echo "Importing file $i"
    pyrit -i $i import_passwords
    if [ $? == "0" ]; then
        rm $i
    else
        pyrit -i $i import_passwords
        if [ $? == "0" ]; then
            rm $i
        else
            echo "File $i seems to be giving Pyrit some issues. Skipping it for now, not removing. Run again later with this file alone.."
        fi
    fi
done

echo "Import complete!"
echo "Split files removed if imported."

[dTechLife]

Another trace

Pyrit 0.5.1 (C) 2008-2011 Lukas Lueg - 2015 John Mora https://github.com/JPaulMora/Pyrit This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/mydb'... connected. 18800000 lines read (91829.6 lines/s)... Traceback (most recent call last): File "/usr/local/bin/pyrit", line 6, in pyrit_cli.Pyrit_CLI().initFromArgv() File "/usr/local/lib/python2.7/dist-packages/pyrit_cli.py", line 118, in initFromArgv func(self, **options) File "/usr/local/lib/python2.7/dist-packages/pyrit_cli.py", line 375, in import_passwords pwstore.store_password(line) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 319, in store_password self._flush_bucket(pw_h1, pw_bucket) File "/usr/local/lib/python2.7/dist-packages/cpyrit/storage.py", line 1153, in _flush_bucket for db_bucket in q.filter(PAW2_DBObject.h1 == pw_h1): File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in iter return self._execute_and_instances(context) File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2776, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute return meth(self, multiparams, params) File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement compiled_sql, distilled_params File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context context) File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception exc_info File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context context) File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute cursor.execute(statement, parameters) File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute self.errorhandler(self, exc, value) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorvalue sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT passwords._key AS passwords__key, passwords.h1 AS passwords_h1, passwords.numElems AS passwords_numElems, passwords.collection_buffer AS passwords_collection_buffer \nFROM passwords \nWHERE passwords.h1 = %s'] [parameters: ('76',)]