Closed GoogleCodeExporter closed 8 years ago
Please try again with r120. The PasswordStore-code was borked in previous
svn-versions.
Some of the storage code is indeed going to get C'ed...
Original comment by lukas.l...@gmail.com
on 12 Jul 2009 at 9:00
It is better, at least in the beginning. I am seeing rates of "178,082",
"154,887",
and "189,570". I am going to leave it running, and recalculate the rate when it
is
done. It seems to pause for a few seconds at a time every so often, and may
slow down
further into the list.
I noticed after I let the previous run finish that the passwords directory was
only
12gb instead of 24gb. Does the method of storage effectively compress the data?
Original comment by starhe...@gmail.com
on 12 Jul 2009 at 11:04
It looks like once it gets deeper into the list this is actually worse. Based on
the numbers I see right now it would take about 31 hours to complete. The new
rate is
something like 26,210.
It starts off fast, and then gets slow.
Original comment by starhe...@gmail.com
on 13 Jul 2009 at 4:33
The problem comes from the fact that the storage-code guarantees uniqueness for
every
single passwords throughout the entire database. That means (in theory) that the
2,283,200,000th password has to be compared against the 2,283,199,999 ones that
were
already imported to prevent duplicates.
This is also the reason why importing gets slower over time - the amount of data
Pyrit as to check against increases... The numbers you see are actually still
*much*
better than what you can expect e.g. from a SQL-database :-)
Yes, the internal file format for passwords uses zlib compression. The
time-advantage
of having to read less data from the (slow) disk and the increased CPU-demand
level
at around the same performance numbers, keeping the advantage of less storage
requirement.
You can experiment with Pyrit's storage code to see if you get better
performance
when dealing with these huge wordlists. Please notice that turning up those
knobs
will increase memory consumption while importing. Modify cpyrit_util.py (r120)
in the
following ways.
1. Increase the maximum number of passwords that get stored in a single file by
modifying 'if len(pw_bucket) >= 20000:' in line 449. A good number would be
50.000
2. Pyrit uses a partial index to speed up the comparision against already-stored
passwords. You may try increasing it's complexity from 2^8 to 2^12. Notice
however
that you need to start with a fresh ~/.pyrit if you do that.
Modify 'h1_list = ["%02.2X" % i for i in xrange(256)]' and change 256 to 4096.
Modify 'pw_h1 = PasswordStore.h1_list[hash(passwd) & 0xFF]' and change 0xFF to
0xFFF
Original comment by lukas.l...@gmail.com
on 13 Jul 2009 at 7:20
I've rewritten the important storage code in C which allows better
multithreading and
added more code to take use of multiple threads reading/writing to the
PasswordStore.
After extensive testing I've decided that this adds too much complexity to the
codebase and is not worth the effort. The performance increase does not justify
more
than ~500 lines of new code and the current stage of the project. So I've
thrown days
of work into the trashbin :-)
We'll have to live with what we got for now.
Original comment by lukas.l...@gmail.com
on 18 Jul 2009 at 3:29
Original issue reported on code.google.com by
starhe...@gmail.com
on 12 Jul 2009 at 7:15