search

zrashwani / arachnid

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
MIT License
253 stars 60 forks source link

fixed #14

Closed howtomakeaturn closed 8 years ago

howtomakeaturn commented 8 years ago

The original base url extract algorithm is buggy, this commit fixes it.

For examlpe, all these urls should have the same base url:

zrashwani commented 8 years ago

thank you, you are right...