search

Samita53 / ldspider

Automatically exported from code.google.com/p/ldspider
0 stars 0 forks source link

Documents dereferenced mulitple times with -n #18

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. crawl with -n
2. does not check seen list whether URI previously seen

Same URI in seed file should only crawled once.

Original issue reported on code.google.com by andr...@harth.org on 2 May 2012 at 9:19