anidata / palantiri

Web crawler to collect data on ht
MIT License
17 stars 8 forks source link

Updated and semi-working continuous crawler #12

Closed bmenn closed 7 years ago

bmenn commented 7 years ago

Made some fixes/updates so that the crawler can run continuous at a regulated pace, so we don't need to setup Tor/Privoxy for it to work.