Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
226
stars
45
forks
source link
Running locally and gzip support - and replacing the hash monkey patch with a static helper. #1
Closed
rojotek closed 12 years ago
Stewart,
It would be awesome if you'd merge these.
Cheers,
Rob