TraversalWorkQueueItem.doWork calls connectorFinishedTraversal too often

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Read the source.

What is the expected output? What do you see instead?

The doWork method calls HostLoadManager.connectorFinishedTraversal when 
runBatch 
processes less than batchHint documents. But the SPI says "the implementor is 
free to use a 
query that only returns a small number of results, if that gets better 
performance." (See the 
TraversalManager doc comment.) I think that the connectorFinishedTraversal 
should only be 
called when runBatch processes zero documents.

What version of the product are you using? On what operating system?

Connector Manager 1.0.3.

Please provide any additional information below.

The Livelink connector happens to return fewer than batchHint documents in many 
cases for 
performance. To improve database performance, we produce batchHint candidates 
and then 
filter them according to the connector instance configuration. Since the 
Connector Manager will 
never process more than batchHint documents, we don't want to waste resources 
by starting with 
more than batchHint candidates, so we often end up returning fewer than 
batchHint results.

Probably a P3 until bug 689553 is fixed, when it may become a P1, in order to 
avoid extraneous 
long pauses in the traversal.

Original issue reported on code.google.com by jl1615@gmail.com on 21 Dec 2007 at 12:36

[GoogleCodeExporter]

Original comment by donald.z...@gmail.com on 18 Apr 2008 at 10:31

• Changed state: Accepted

[GoogleCodeExporter]

r781 | mgronber | 2008-05-02 18:15:41 -0700 (Fri, 02 May 2008) | 63 lines

Fix for Bug 689553 (Connector Manager should pause periodically and retry
interval should be configurable via the Admin Console), Issue 66
(TraversalWorkQueueItem.doWork calls connectorFinishedTraversal too often), and
Issue 67 (Scheduler runs traversals faster than configured traversal rate).

The default retry delay between times when the runBatch() returns no documents
and the next time the schedule is tested has been changed from 0 to 5 minutes.

The fix for Issue 66 involved fixing the conditional that TWQI.doWork() method
was using to determine if the traversal was finished.

For Issue 67 it turned out that there was a problem with the
TraversalScheduler's use of the TraversalWorkQueueItem.  I was not involved with
the design, however, it appears it is designed so only one WorkQueueThread was
suppose to operate on a given TWQI at a time.  In the run() method, when the
TWQI was added to the WorkQueue after the first run, it's isFinished field was
incorrectly left as true.  This would cause the subsequent call to
TWQI.getNumDocsTraversed() to return right away with 0 rather than waiting until
the doWork() was finished.  This would be recorded as a failure() and the run()
loop would continue.  In the mean time, the TWQI added to the WorkQueue was
being processed by a WorkQueueThread.  After the failure in the run() method it
would restart the loop and incorrectly add the TWQI back on the WorkQueue
causing it to be picked up by another thread.

This might also be related to the issue where crawl rates seem to decay after
some time.  The subsequent (false) failures will cause the TraversalScheduler to
backoff, thinking the Connector is having problems.

Finally, the TraversalScheduler was not able to differentiate between the case
where the traversal batch was complete and the case where it was just throttled
back because of host load.  I have fixed this so now it stays aggressive when
the host load was exceeded and only observes the retry delay when it seems the
batch has been completed.

ChangeLog:

*
projects/connector-manager/source/java/com/google/enterprise/connector/servlet/S
etSchedule.java:
  - Changed default retryDelayMillis from 0 to (5 * 60 * 1000) 5 minutes.

*
projects/connector-manager/source/java/com/google/enterprise/connector/common/Wo
rkQueue.java:
  - Added some FINEST logging messages to monitor work being added and removed
    from the WorkQueue and create toString() method on the TWQI to provide
    detail to the log.

*
projects/connector-manager/source/java/com/google/enterprise/connector/scheduler
/TraversalScheduler.java:
  - Removed unused local to remove warning.
  - Fix for Issue 67 where the work item was being added with a state already
    finished causing multiple threads.
  - Related fix where incorrect condition used to determine failure case.
  - Added batchHint field to TraversalWorkQueueItem to support correct
    conditional check for both finished and failure.
  - Changed logging in doWork() from FINER to FINEST since used for debugging,
    added batchHint to log, and moved log statements around actual call to 
    runBatch().
  - Fix for Issue 66 where connectorFinishedTraversal() was being called
    incorrectly for those cases where the Connector was returning less documents
    than the recommended batchHint. 

*
projects/connector-manager/source/java/com/google/enterprise/connector/scheduler
/HostLoadManager.java:
  - Added Logger.
  - Added FINEST level logging to determineBatchHint() to aid in debugging.

Original comment by mgron...@gmail.com on 3 May 2008 at 1:21

• Changed state: Fixed

[GoogleCodeExporter]

Original comment by mgron...@gmail.com on 19 Jun 2008 at 6:35

• Added labels: Milestone-Release_1.1.0