search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
226 stars 45 forks source link

Fixed bugs -- please merge. #6

Closed rojotek closed 12 years ago

rojotek commented 12 years ago

normalize adressable uris before sending them as a part of a http get or head request. moved version back to it's own file -- needed for the use of local filesystem references with bundle. fix bug where Stats.get_statistics was being used. fixes around redirect locations - particularly for head requests.

rojotek commented 12 years ago

included an extra series of commits.

including: 1) moving version back to it's own file -- needed for the use of local filesystem references with bundle. 2) fix bug where Stats.get_statistics was being used. 3) fixes around redirect locations - particularly for head requests.