Improve traversal: Scan web sites, look for new lists as required and not in every batchTraversal

[GoogleCodeExporter]

Current batch traversal:
Checks for new lists every time
Scans all webstates (sites) to see if docs are pending from previous crawl 
cycle

The above should be handled intelligently such the # of web service calls are 
reduced and updates are being fetched at consistent rate. This should improve 
the connector's traversal performance

Original issue reported on code.google.com by rakeshs101981@gmail.com on 25 Nov 2009 at 1:40

[GoogleCodeExporter]

This issue is filed as Google issue #6513912

Original comment by tdnguyen@google.com on 17 May 2012 at 11:53

• Changed state: Duplicate
• Added labels: ****
• Removed labels: ****