search

digininja / pipal

Pipal, THE password analyser
www.digininja.org/projects/pipal.php
630 stars 123 forks source link

Pipal lag and failure to complete analysis (RAM limitations?) #42

Closed tokyoneon closed 6 years ago

tokyoneon commented 6 years ago

Hey @digininja , I would like to feature pipal on Null Byte, but I'm experiencing an issue.

The wordlist being analyzed is 92,400,000 lines, 1.1Gb. Pipal seems to lag after ~2 hours of processing. At that point my CPU usage drops down to normal levels and the ETA starts to slowly increase until it stops. This was done in a Kali VM with 4 cores and 6Gb RAM. Tried again on a dedicated Ubuntu machine with an i7 and 16Gb RAM -- same issue.

Does pipal have limitations? What might be causing the lag and how can I debug?

EDIT:

I just realized Ruby v1.9 is recommended in the README.

> apt-cache policy ruby
ruby:
  Installed: 1:2.5.0
  Candidate: 1:2.5.0
  Version table:
 *** 1:2.5.0 500
    500 http://http.kali.org/kali kali-rolling/main amd64 Packages
    100 /var/lib/dpkg/status

But pipal works fine when analyzing smaller wordlists using v1.2. Should I try manually installing v1.9.x in Kali? There's an APT candidate for v2.3 and v2.5, will those suffice?

P.S. I'm not a ruby coder

digininja commented 6 years ago

The readme is out of date, 1.9 was recommended over 1.87.

There is an issue with memory usage as it caches the results as it goes through so with such a big list it will be eating a lot of memory.

I've got a rewrite that isn't release yet that stores the results on disk rather than in memory and runs quite a bit faster. I should try to get the finished off and released. Till then, try using just one checker at a time, that will reduce the memory usage.

On Sat, 7 Apr 2018, 22:30 tokyoneon, notifications@github.com wrote:

Hey @digininja https://github.com/digininja , I would like to feature pipal on Null Byte https://creator.wonderhowto.com/tokyoneon/, but I'm experiencing an issue.

The wordlist being analyzed is 92,400,000 lines, 1.1Gb. Pipal seems to lag after ~2 hours of processing. At that point my CPU usage drops down to normal levels and the ETA starts to slowly increase until it stops. This was done in a Kali VM with 4 cores and 6Gb RAM. Tried again on a dedicated Ubuntu machine with an i7 and 16Gb RAM -- same issue.

Does pipal have limitations? What might be causing the lag and how can I debug?

EDIT:

I just realized Ruby v1.9 is recommended in the README.

apt-cache policy ruby ruby: Installed: 1:2.5.0 Candidate: 1:2.5.0 Version table: *** 1:2.5.0 500 500 http://http.kali.org/kali kali-rolling/main amd64 Packages 100 /var/lib/dpkg/status

But pipal works fine when analyzing smaller wordlists using v1.2. Should I try manually installing v1.9.x in Kali? There's an APT candidate for v2.3 and v2.5, will those suffice?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/digininja/pipal/issues/42, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWYZUOH6RgLuXVwsmx62TuEV3sWqdks5tmS_ngaJpZM4TLQt7 .

tokyoneon commented 6 years ago

Thanks for the reply. I think the basic.rb checker is the only one enabled.

Do you have an estimated time frame for when the rewrite will be uploaded to GitHub? I wanted to publish my article ~Tuesday but I might be able to postpone it a few days. I would really like to analyze this large wordlist and feature it in the article.

digininja commented 6 years ago

It won't be this week. Would you be able to send me the list? I'll run it through and send you the results, would be a good test.

On Sat, 7 Apr 2018, 22:52 tokyoneon, notifications@github.com wrote:

Thanks for the reply. I think the basic.rb checker is the only one enabled.

Do you have an estimated time frame for when the rewrite will be uploaded to GitHub? I wanted to publish my article ~Tuesday but I might be able to postpone it a few days. I would really like to analyze this large wordlist and feature it in the article.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/digininja/pipal/issues/42#issuecomment-379501826, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWR-nTAKCv_efubB9f7tNdjbnUemPks5tmTUSgaJpZM4TLQt7 .

tokyoneon commented 6 years ago

-----BEGIN PGP MESSAGE----- Version: GnuPG v2

hQEMA3E/TAUfwjjMAQf/cVlnFK8SwdHmrKPkDYnpNveAxO2hSCvo4xlcYpl7DMUL wwrZZn4r7gFmz6L9Gex3kMrVlKRnyxH1hfsAVMBhy2pIi8AUq+GK8pF2Qx1h4dxO inBIccCUQJUh1qMRDsYg5/IAH2Zih58qjvFZCwA7fczEenbBmrPtZA0V1WgwJnHY XmLo3w7xbq1ivkSoMxFc733oFY1ez9etS8jsgKwlvQoJaeGJHmfFhVRLkVseAYg/ MhKHWKwn9IWRq6vCrz5HXh35Rdx3wNY6UIF7R7lgEuK3jlJoI3W+8QCOwGLcqdyk yBKD+A68GVtngNRAx2dt1Zx+WKhmY98+zVcKUYUjGIUCDAOFGPumG2fL5gEP/0E4 xM7w6yyNv4MVXXUFfdydCti15L9D4MaZ6SEiUnYkrscj70uCxXWFSMPNwOXYOrir NeNSwP2j9NgqDy1270q3D0vOG8N6N1MXQeta8nKRm2nAW7qRjVOqnv20fdv4jkue NPN1Yx3d6S0sKLESnDRJOPNQLppO/OV4hSND4Rr1reslJ0eBE7fmsRM0qsXngc1V xuBBsrJg3qdNZjTJ4N/IgcHEaKrlt1QdsRKVmt8CZveFYqvvb7zblfEcjnUh3+95 VVhdSX9cl4M7XNVjWwXGqtgLvCpjhx+YgMVLdAOG0aUY7m4nEL5uh5WlODmzT86F ZRjK/8tgTNpFG34uJudfEmj4SQrEMiH5ul/fcJjh59HqhlBb09UBsk8m5fiZWL8D MUJD5xFPBblm7p7lDQ3JOaXabROG5FvYRemS34vFLBnH8eLJRgS4ok/GUG+NrSbv VE8nELxTUt5pWQNafCG7CyCPaiuFvEOjj70tpQV4/GodUf7nKcezYetsk4H/POaw mgK39MYpC+nMl7hB4skjWGkcIX4OodpTcGW6sDv+ix+NhT9GnfgCr0u/4hYaB5CC wDUjpPnAQ+4NfXrq567RZUdLj8FNmyiDDaSad+PDRsnGU/LcwsXLEu54BO11F3qX E4VQFf8pJvpBG/Q3gW5gQF15G2os+fqxcsjg7A7b0ukB6XgMaRqYM4HQKdUjngfZ L59+jSr4GrJ6KUSzFApvzNxpBA97aqpyDhiUBBAAGhPqOxRuNqUJnk2/2Ch5TEZa Tf+QHeGwPS+aKU1Wle3ZnfK/DP0lhES9zpesl/PMAxRrRi46ocq3E0rDYnSneAfr duO+f92MoBh56uvdhsGkkaub5Qa8OTO3XO42txZvvx52QyvQgZuXRTar8dbijqlX LoLdxSP5QzdIDmJxG0/oZ8zsPtry+vzULcbeoWK6LJPZunolUl9EZ9178re28H20 Sh/f0wDQWPBFzW/NclEb1zOfKkfW7D4uujUBeZSsqyIHOut/DJjJYlVaqdb8KAYN Crma6TTkLXBryzcY6unzhHPg/6Sy1BrZqs9L8LAWPNfUY+ltuzfn23OypmYj7i1z aOgZzjiwyiP457AI37NOKkWUqNeNS4WOZSxrzPcQHcOnNUPl6u1lP+N/gneI/90K lcu/F/P9NYT/ItDU4AR294aq3JmR40pMh50KFM/JzeApymA6zwLcj8gx7H2NqTxd eBwq7ERyzXFOpnsfvJjXeV1yWqv0ZrSXkZX1ZwjmI0zi1A4selZXzNWHbnJuJGQD UdyfjryZNxelcjYagY9kWVYpCJxU8ZL3Dp8hpbHsepNNZ0l1Ubf1Eq8+zpijgjmg jVwdlidNdimnc4xXsbb+1TzgsaHmXau3oeY2uqIST9W6hXIpMLk1CJV1Xkx8hqfc 2J+KD2zRcuORdWBk/jYR0fQw6wNdIGRBC92c37c= =dbMz -----END PGP MESSAGE-----

tokyoneon commented 6 years ago

Thanks for your help btw. Think you can analyze it this weekend?

digininja commented 6 years ago

Unfortunately I'm currently sat in a hotel about to run a marathon so might not be home in time tonight to do it.

Where did you get the PGP key? If it was keybase I can read your message, if not, try again and I'll go out and try to kill the key wherever it is.

On Sun, 8 Apr 2018, 02:59 tokyoneon, notifications@github.com wrote:

Reopened #42 https://github.com/digininja/pipal/issues/42.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/digininja/pipal/issues/42#event-1562415219, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWe2JOdvM8-X72SfROeiAkdGi_I6aks5tmW72gaJpZM4TLQt7 .

tokyoneon commented 6 years ago

Yup, Keybase. I did notice an older key on the MIT database, but figured it was revoked/expired. 

    > gpg2 --fingerprint digininja
pub   rsa4096/E447F6DB 2015-03-02 [SCEA]
      Key fingerprint = 79D0 248B 504B 5B30 6911  7DEF D671 D402 E447 F6DB
uid         [ unknown] keybase.io/digininja <digininja@keybase.io>
sub   rsa2048/6236EB96 2015-03-02 [S] [expires: 2023-02-28]
sub   rsa2048/1FC238CC 2015-03-02 [E] [expires: 2023-02-28]

I think I'll postpone the article till Thursday if you don't get to analyzing the database this weekend. The article will be more interesting for it. So it's worth the wait :+1:

tokyoneon commented 6 years ago

Oh, if possible, get the --top 500 and enable the below checkers?

-rw-r--r-- 1 root root  11K Apr  7 14:56 basic.rb
-rw-r--r-- 1 root root 1.6K Apr  7 14:56 EN_religion_checker.rb
-rw-r--r-- 1 root root 1.2K Apr  7 14:56 EN_military_checker.rb
-rw-r--r-- 1 root root 1.4K Apr  7 14:56 EN_family_checker.rb
-rw-r--r-- 1 root root 1.5K Apr  7 14:56 EN_explicit_checker.rb
-rw-r--r-- 1 root root 1.1K Apr  7 14:56 EN_emotion_checker.rb
-rw-r--r-- 1 root root  510 Apr  7 14:56 EN_colour_checker.rb
-rw-r--r-- 1 root root  757 Apr  7 14:56 EN_violence_checker.rb
-rw-r--r-- 1 root root  758 Apr  7 14:56 EN_vehicle_checker.rb
-rw-r--r-- 1 root root  825 Apr  7 14:56 EN_sport_checker.rb
-rw-r--r-- 1 root root 1000 Apr  7 14:56 special_checker.rb
tokyoneon commented 6 years ago

Hey, really sorry, looks like I deleted the keybase app from my other computer. I can't view the messages you sent me. I would have to reset my entire Keybase account to regain access Keybase's chat functionality, but doing so would still wipe my previous message history.

Would you be able to send me PGP blocks here or via email? Really sorry for the inconvenience. #EncryptionFail

digininja commented 6 years ago

Ye sure, will send it over in some way.

Not all the checkers are working yet but I'll get as many going as I can.

On Mon, 9 Apr 2018, 02:20 tokyoneon, notifications@github.com wrote:

Hey, really sorry, looks like I deleted the keybase app from my other computer. I can't view the messages you sent me. I would have to reset my entire Keybase account to regain access Keybase's chat functionality, but doing so would still wipe my previous message history.

Would you be able to send me PGP blocks here or via email? Really sorry for the inconvenience. #EncryptionFail

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/digininja/pipal/issues/42#issuecomment-379600935, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWSXQF0ZAqgIy1sgPEJUyLh_6tv1Wks5tmrdwgaJpZM4TLQt7 .