Closed heynemann closed 10 years ago
If the crawler starts crawling a page and it's not in the root of a domain:
http://www.page.com/something/else.html
And it finds links like:
/other/test.html
It translates to http://www.page.com/something/other/test.html, when it should be http://www.page.com/other/test.html.
If the crawler starts crawling a page and it's not in the root of a domain:
http://www.page.com/something/else.html
And it finds links like:
/other/test.html
It translates to http://www.page.com/something/other/test.html, when it should be http://www.page.com/other/test.html.