search

openwall / john-tests

Test Suite for John the Ripper
24 stars 15 forks source link

1 GB systems fail due to use of unique (and it's recent large defaults) #64

Closed magnumripper closed 9 years ago

magnumripper commented 9 years ago

See https://github.com/magnumripper/JohnTheRipper/issues/1872

Easy fix is just adding -mem option to unique call, but I'll look into dropping use of unique instead. No need to use anything external for that.

magnumripper commented 9 years ago

I opted to just add the -mem option. The code is horrible, calling external programs like it were a bash script. I don't have the energy to rewrite it right now.

magnumripper commented 9 years ago 
$ time ../run/unique <rockyou.original.dlst rockyou.lst
Total lines read 32603388 Unique lines written 14344391

real    0m10.261s
user    0m9.750s
sys 0m0.479s

$ time perl -ne 'print if !$u{$_}++' < rockyou.original.dlst > rockyou.lst

real    0m27.324s
user    0m26.008s
sys 0m1.247s

OK, perl is slower but that's a huge file and for jtrts.pl we're only handling kilobytes anyway.

magnumripper commented 9 years ago

And BTW using the decreased -mem, unique is not much faster

$ time ../run/unique -mem=20 <rockyou.original.dlst deleteme3
Total lines read 32603388 Unique lines written 14344391

real    0m23.846s
user    0m23.296s
sys 0m0.483s