Using Tor for spiders - Githubissues

vipulgupta2048 / scrape

Mission - scraping the planet, one website at a time

MIT License

10 stars 13 forks source link

Using Tor for spiders #60

Closed atb00ker closed 6 years ago

atb00ker commented 6 years ago

Using of Tor seems important for the newsx spider, I have tried throttle it as much as possible but it wouldn't work! Whoever finds time, please assign yourself the task! :)

thisisayush commented 6 years ago

Fixed in commit 697f9bad7d81435edd44958c80ffd9382e9a571f