kgretzky / dcrawl

Simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.
MIT License
516 stars 93 forks source link