search

kgretzky / dcrawl

Simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.
MIT License
516 stars 93 forks source link

Suggestion #6

Open mathieu-aubin opened 5 years ago

mathieu-aubin commented 5 years ago

It would be nice to be able to choose User-Agent or, even better and not so complicated either, to create a random user agent for each requests. Something like they did with Go-Colly

Check it out at https://github.com/gocolly/colly/blob/master/extensions/random_user_agent.go

ubogdan commented 5 years ago

This require some code rewrite but it seems this repo is dead.