import_passwords fails on large file with MySQL database

Hi there!

I'm attempting to import a large (1.3GB) text file full of passwords into 
Pyrit's MySQL database and it has failed repeatedly.

The file in question is very high-quality (UNIQPASS v11, 
https://dazzlepod.com/uniqpass/) so I'm confident that it's not causing the 
issue.  I'm using a MySQL database as my default storage as configured with 
~/.pyrit/config.

My system is a quad-core AMD CPU running 64-bit Ubuntu 12.10.

When I run <pyrit -i UNIQPASS.txt import_passwords> it runs for some variable 
length of time before crashing.  (Same issue is present in a detached screen 
session as in a remote shell.)

Strange thing is, it produces a different error and crashes after a different 
number of lines each time I run it (asterisks are mine for readability):

****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
44700000 lines read (567195.0 lines/s)... Segmentation fault (core dumped)
*****************************************************************************
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
97100000 lines read (114239.9 lines/s)... Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 115, in initFromArgv
    func(self, **options)
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 355, in import_passwords   
    pwstore.store_password(line)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 363, in store_password
    self._flush_bucket(pw_h1, pw_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 1196, in _flush_bucket
    bucket.difference_update(db_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 926, in __iter__
    self.collection.unpack(self.collection_buffer)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 287, in unpack
    raise DigestError("Digest check failed")
cpyrit.storage.DigestError: Digest check failed
****************************************************************************
Connecting to storage at 'mysql://localhost/pyrit1'...  connected.
45000000 lines read (459279.8 lines/s)... Traceback (most recent call last):
  File "/usr/bin/pyrit", line 6, in <module>
    pyrit_cli.Pyrit_CLI().initFromArgv()
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 115, in initFromArgv
    func(self, **options)
  File "/usr/lib/pymodules/python2.7/pyrit_cli.py", line 355, in import_passwords   
    pwstore.store_password(line)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 363, in store_password
    self._flush_bucket(pw_h1, pw_bucket)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 1199, in _flush_bucket
    session.add(PAW2_DBObject(pw_h1, bucket))
  File "<string>", line 4, in __init__
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 98, in initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 917, in __init__
    key, collection_buffer = PAW2_Buffer(self.collection).pack()
  File "/usr/lib/pymodules/python2.7/cpyrit/storage.py", line 276, in pack
    b = zlib.compress('\n'.join(self.collection), 1)
TypeError: sequence item 9425: expected string,  found
****************************************************************************

Considering that Pyrit is supposed to be a robust and scalable tool that can 
manage massive databases, I hope this is just me! Please let me know if there 
is anything I can do to help troubleshoot or if I'm missing something.

Thanks for making a great tool.
Alex
Original issue reported on code.google.com by iAlex.R...@gmail.com on 14 Feb 2013 at 7:04
oohave / pyrit

import_passwords fails on large file with MySQL database #418
