Graph crawling indexes the index page twice

opencharles / charles

Java web crawling library

BSD 3-Clause "New" or "Revised" License

32 stars 9 forks source link

Graph crawling indexes the index page twice #97

Open amihaiemil opened 7 years ago

amihaiemil commented 7 years ago

Both www.domain.com/index.html and www.domain.com are indexed as different documents, resulting in duplicate search results (the index page is indexed twice)