only try and crawl links on a page 1 time.

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.

MIT License

226 stars 45 forks source link

only try and crawl links on a page 1 time. #12

Closed rojotek closed 11 years ago

rojotek commented 11 years ago

Hey Stew,

A minor pull request for you -- it does a uniq on the list of internal links to try and prevent pages from being added to the crawl set more than once.

Rob

travisbot commented 11 years ago

This pull request passes (merged 4c5cf374 into e977a70e).