search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

Issue 29: Normalize URLs when storing inbound links #30

Closed gh2k closed 9 years ago

gh2k commented 10 years ago

See: https://github.com/stewartmckee/cobweb/issues/29

I found that when scraping my site, I didn't see inbound links for URLs that had been changed by the normalization process.

stewartmckee commented 9 years ago

Sorry took so long... but merged now. Thanks.