search

digininja / CeWL

CeWL is a Custom Word List Generator
2.01k stars 262 forks source link

Issue with CeWL "grinding"? #40

Closed 7MinSec closed 6 years ago

7MinSec commented 6 years ago

Hi there,

I'm running the latest CeWL on a fresh Kali VM, and asked it to do a pretty light enumeration of a site (depth of 1 and minimum character count of 7). CeWL will just grind away for hours and never finish (and never write anything to the output .txt file). It also seems to drag down performance of the VM and make it unresponsive (even though top doesn't seem to indicate the ruby is crushing the VM). CeWL is the only thing running on the VM.

Curious if you've ever seen this before or have any troubleshooting ideas to try?

Thanks, Brian

digininja commented 6 years ago

Not something I've noticed before, can you share the URL?

On Wed, 27 Jun 2018, 18:19 Brian Johnson, notifications@github.com wrote:

Hi there,

I'm running the latest CeWL on a fresh Kali VM, and asked it to do a pretty light enumeration of a site (depth of 1 and minimum character count of 7). CeWL will just grind away for hours and never finish (and never write anything to the output .txt file). It also seems to drag down performance of the VM and make it unresponsive (even though top doesn't seem to indicate the ruby is crushing the VM). CeWL is the only thing running on the VM.

Curious if you've ever seen this before or have any troubleshooting ideas to try?

Thanks, Brian

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/digininja/CeWL/issues/40, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWXloaok4aYrM3ZqobJlgFsxgCTdyks5uA763gaJpZM4U6Ffd .

7MinSec commented 6 years ago

No, but let me try from another Kali machine next week and see if it's "just a me thing." Thanks.

digininja commented 6 years ago

Try running the latest GitHub version as that is the easiest for me to debug is there is a problem.

On Fri, 29 Jun 2018, 18:39 Brian Johnson, notifications@github.com wrote:

No, but let me try from another Kali machine next week and see if it's "just a me thing." Thanks.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/digininja/CeWL/issues/40#issuecomment-401424229, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWeIzXU3JyXchC3p_BrIudFX4ytwkks5uBmZngaJpZM4U6Ffd .

7MinSec commented 6 years ago

Hi, I turned on verbose logging and found that CeWL seemed to stop crawling the site about 20 minutes into operation. However, upon subsequent runs it gets stuck again - but in a different place. I have a feeling something on the client side is stepping in and throttling/blocking requests. So I'm pretty sure it's a site-specific issue, thus I'm closing this thread. Thanks.