search

wishlu / milkshake

I drink your milkshake.
0 stars 0 forks source link

Distributed Web Crawler / Spider / Scraper #13

Open tylerjharden opened 9 years ago

tylerjharden commented 9 years ago

Part of Blender milestone.

Bookmark: http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/

tylerjharden commented 9 years ago

Dotmic is a somewhat poor site that already does this, and explains how they do it here: Bookmark: http://www.dotmic.com/about/