Allow GSA to request log files from connector manager

[GoogleCodeExporter]

This is a feature request to allow the GSA to send an HTTP request to the
connector manager and retrieve all log files.

This would help Google support to diagnose problems more quickly, because
they could get the error logs from the connector while logged in to the
GSA. Currently, support will often do a Webex system to see the logs which
is very inefficient.

For security reasons, the connector manager could restrict this URL to the
appliance, since it knows the IP address of the appliance.

Original issue reported on code.google.com by jlo...@gmail.com on 21 Mar 2008 at 9:51

[GoogleCodeExporter]

Original comment by mobe...@gmail.com on 22 Mar 2008 at 12:45

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Original comment by mobe...@gmail.com on 18 Apr 2008 at 9:36

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Original comment by mobe...@gmail.com on 18 Apr 2008 at 10:31

[GoogleCodeExporter]

Original comment by Brett.Mi...@gmail.com on 25 May 2008 at 2:21

• Changed state: Accepted

[GoogleCodeExporter]

Fixed in revision r832.

This series of changes allows the Connector Manager's connector logs and 
feed logs to be fetched via a servlet interface.  The servlet allows the 
user to list available log files, fetch individual log files, and fetch a
ZIP archive of all the log files.  

At this time the user may retrieve connector log files, feed log files, 
and teed feed files.  Access to this servlet is restricted to either
localhost or gsa.feed.host, based upon the HTTP RemoteAddress.

Usage:
-----
To list the available connector log files:
  http://[cm_host_addr]/connector-manager/getConnectorLogs

To view an individual connector log file:
  http://[cm_host_addr]/connector-manager/getConnectorLogs/[log_file_name]
where [log_file_name] is the name of one log files returned by the list.
For instance, 'google-connectors.otex0.log'.  As a convenience, the log 
name may be simply the log file generation number, '0' in the above example,
and it gets automatically expanded.

To retrieve a ZIP archive of all the connector log files:
  http://[cm_host_addr]/connector-manager/getConnectorLogs/*
or
  http://[cm_host_addr]/connector-manager/getConnectorLogs/ALL

To list the available feed log files:
  http://[cm_host_addr]/connector-manager/getFeedLogs

To view an individual feed log file:
  http://[cm_host_addr]/connector-manager/getFeedLogs/[log_file_name]
where [log_file_name] is the name of one log files returned by the list.
For instance, 'google-connectors.feed0.log'.  As a convenience, the log 
name may be simply the log file generation number, '0' in the above example,
and it gets automatically expanded.

To retrieve a ZIP archive of all the feed log files:
  http://[cm_host_addr]/connector-manager/getFeedLogs/*
or
  http://[cm_host_addr]/connector-manager/getFeedLogs/ALL

To list the name and size of the teed feed file:
  http://[cm_host_addr]/connector-manager/getTeedFeedFile

To view the teed feed file:
  http://[cm_host_addr]/connector-manager/getTeedFeedFile/[teed_feed_name]
where [teed_feed_name] is the base filename of the teed feed file.
WARNING: The teed feed file can be HUGE.  It is suggested you either
request a managable byte range (see below) or fetch the ZIP archive file
(which may still be HUGE).

To retrieve a ZIP archive of the teed feed file:
  http://[cm_host_addr]/connector-manager/getTeedFeedFile/*
or
  http://[cm_host_addr]/connector-manager/getTeedFeedFile/ALL
or
  http://[cm_host_addr]/connector-manager/getTeedFeed/[teed_feed_name].zip
where [teed_feed_name] is the filename of the teed feed file.

Byte Range Support:
------------------
This servet supports a subset of the RFC 2616 byte range specification
to retrieve portions of the log files.  Since the connector logs are
50MB each and the teedFeedFile can be gigabytes, requesting a portion
of the log may be prudent.  This servlet supports byte range specifier
in either the HTTP Range: header or in the Query fragment of the request.
For instance:
  http://[cm_host_addr]/connector-manager/getFeedLogs/0?bytes=0-1000
returns the first 1001 bytes of the current feed log.

  http://[cm_host_addr]/connector-manager/getFeedLogs/0?bytes=-1000
returns the last 1000 bytes (the tail) of the current feed log.

  http://[cm_host_addr]/connector-manager/getFeedLogs/0?bytes=1000-
returns everything after the first 1000 bytes of the current feed log.

Multipart byte ranges are NOT supported (ie bytes=0-100,1000-2000).
Byte range requests for log listing pages and ZIP archive files are
ignored.

Redirects and curl:
------------------
When using shorthand file specifications, like generation numbers,
'ALL', or '*', this servlet returns a redirect to the actual filename.
This allows the browser, wget, or curl to pull the true filename off
the redirected URL so that it can name the file when storing locally.
If using curl to retrieve the files, please to use 'curl -L' to tell
curl to follow the redirect.  Unfortunately, when using 'curl -O' to
save the file locally, curl uses the pre-redirected name, rather than
the post-redirected name when naming the local file.  This forces you
to use 'curl -L -o output_filename' anyway, so you might as well 
specify the full filename in the URL to begin with.

Wget handles redirects appropriately without intervention and names
the saved file as expected.

Compressed Content-Encodings:
----------------------------
When serving up individual logs, this servlet supports compressing
the output stream using the gzip or deflate Content-Encodings, if 
the client accepts them.  When using curl, you can enable compressed
Content-Encoding by specifying 'curl --compressed ...'.

Upgrading existing Connector Manager installations:
--------------------------------------------------
Note that since this change modifies both the Connector Manager 
applicationContext.xml and its web.xml, upgrading existing CM
installations is more difficult then simply dropping in a new
jar file and restarting Tomcat.  These XML files must be updated
in existing deployments as well.  This might be as simple as 
replacing the existing deployed version with the new version,
however beware of overwriting customer modifications to these
files.  In my case, I also had to update my Connector Manager
applicationContext.properties file to include the feedLoggingLevel
property.

Change Log:
----------

A 
projects/connector-manager/source/java/com/google/enterprise/connector/servlet/G
etConnectorLogs.java
   - New class that implements the log fetching capability servlet.

M 
projects/connector-manager/source/java/com/google/enterprise/connector/servlet/S
ervletUtil.java
   - Added additional XML tags and MIME types used when fetching logs.
   - Added a debugging routine that prints the servlet request to the response.

M 
projects/connector-manager/source/java/com/google/enterprise/connector/manager/C
ontext.java
   - Added an method to fetch the configured GSA Feed Host property.

A 
projects/connector-manager/source/java/com/google/enterprise/connector/pusher/Fe
edFileHandler.java
   - A new trivial subclass of java.util.logging.FileHandler that allows the
     retrieval of the pattern string used to generate the log file names.

M  projects/connector-manager/etc/applicationContext.xml
   - Changed the FeedHandler bean instantiator to construct a FeedFileHandler
     object, rather than a java.util.logging.FileHandler object.

M  projects/connector-manager/source/webdocs/web.xml
   - Added the servlet interfaces for retrieving the logs.

Original comment by Brett.Mi...@gmail.com on 19 Jun 2008 at 9:39

• Changed state: Fixed

[GoogleCodeExporter]

Original comment by mgron...@gmail.com on 20 Jun 2008 at 4:34

• Added labels: Milestone-Release_1.1.0

[GoogleCodeExporter]

Minor issues were fixed in r862

*Brett.Michael.Johnson/fix_issue_70_1@862 | Brett.Michael.Johnson | 2008-06-30
16:33:00 -0800 (Mon, 30 Jun 2008)

Description:

This small change corrects a couple of problems discovered in the
recently added GetConnectorLogs implementation.  Specifically,
 - inadequate detection of localhost as the caller address.
 - poor support for platform-specific filename separatorChars in
   the FileHandler pattern.

Change Log:
----------
M
projects/connector-manager/source/java/com/google/enterprise/connector/servlet/G
etConnectorLogs.java
 - added support for File.separatorChar in baseName() and directoryName().
 - allowedRemoteAddr() uses java.net.InetAddr.isLoopbackAddress() for
   better detection of localhost.

Original comment by Brett.Mi...@gmail.com on 30 Jun 2008 at 11:35