search

PILLUTLAAVINASH / google-enterprise-connector-manager

Automatically exported from code.google.com/p/google-enterprise-connector-manager
0 stars 0 forks source link

use /getbacklogcount to do better throttling and monitoring #124

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
See internal issue #1615117 for more information.

Original issue reported on code.google.com by mobe...@gmail.com on 13 Feb 2009 at 7:32

GoogleCodeExporter commented 8 years ago

Original comment by jeffreyl...@gmail.com on 13 Feb 2009 at 7:33

GoogleCodeExporter commented 8 years ago 
From Google bug #1615117:

"Currently feeds are sent to http://<hostname>:19900/xmlfeed and the new url is
http://<hostname>:19900/getbacklogcount . It just returns an integer (count of 
total
unprocessed feed files). Feed generating code (including connectors) can use 
this to
throttle submission rate and this can help us prevent numerous bugs where too 
many
feed files are sent and disk gets full,etc..etc.."

The new URL is not yet available in a released version of the GSA software.

Original comment by jl1615@gmail.com on 13 Feb 2009 at 8:45

GoogleCodeExporter commented 8 years ago 
Basically set a ceiling and floor value for the backlog.  If the backlog 
exceeds the ceiling, pause the 
Scheduler (either in the Scheduler or the HostLoadManager - the latter sounds 
like it should be in charge 
here).  Do not resume feeding until the backlog drops below the floor.

When the feed falls over, we have seen a backlog of 100,000+ feed items, so I 
would set a floor of 10,000 
and a ceiling of 50,000.

Note that setManagerConfig() would have to notify the backlog checker if the 
feed host changes.

Original comment by Brett.Mi...@gmail.com on 10 Apr 2009 at 6:08

GoogleCodeExporter commented 8 years ago 
Tested in 5.2.0.G28, and it works:

HTTP/1.0 200 OK
Content-Type: text/plain
Date: Fri, 10 Apr 2009 20:28:08 GMT
Server: feedergate_1.0
Connection: Close
Content-Length: 1

7

Original comment by jl1615@gmail.com on 10 Apr 2009 at 8:29

GoogleCodeExporter commented 8 years ago

Original comment by mgron...@gmail.com on 6 May 2009 at 9:52

GoogleCodeExporter commented 8 years ago

Original comment by Brett.Mi...@gmail.com on 16 Jul 2009 at 6:31

GoogleCodeExporter commented 8 years ago 
Fixed 07 August 2009 in Connector Manager revision r2196

This set of changes suspends the feed if the GSA is backlogged processing
feed items.  This is used to throttle back the document feed if the GSA has
fallen behind processing outstanding feed items.  We periodically poll the
GSA, asking for the count of unprocessed feed items (the backlog count).
We also define a ceiling and a floor for the backlog count.  If the
backlog count exceeds the ceiling we pause the feed.  We resume the
feed once the backlog count drops down below the floor value.

The floor, ceiling, and poll interval are adminstrator-configurable
by editing the FeedBacklogFloor, FeedBacklogCeiling, and
FeedBacklogCheckIntervalSeconds bean definitions in applicationContext.xml.

This change also add support for the new GSA disk full error (actually
any feed error), suspending the feed until the error condition clears.

Note that early GSAs do not support the getbacklogcount servlet.
This code handles the condition as UnsupportedOperation, and disables
the polling of the backlog count.

Original comment by Brett.Mi...@gmail.com on 7 Aug 2009 at 9:59

GoogleCodeExporter commented 8 years ago

Original comment by jl1615@gmail.com on 27 Oct 2009 at 11:05