search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
226 stars 45 forks source link

Use base url #15

Closed rojotek closed 11 years ago

rojotek commented 11 years ago

Updated the content link parser to use the base url if it is set.