Fix config storage - Githubissues

GoogleCodeExporter commented 8 years ago

checkpoint especially, remove the dependency on Windows registry,
consolidate the location of config files

Original issue reported on code.google.com by jeffreyl...@gmail.com on 21 Apr 2008 at 8:52

GoogleCodeExporter commented 8 years ago

Original comment by mgron...@gmail.com on 18 Jul 2008 at 9:17

Added labels: Priority-Medium
Removed labels: Priority-Low

GoogleCodeExporter commented 8 years ago

In a closely related matter, isolate the connector properties file code to
InstanceInfo.  (Currently InstanceMap knows about the properties file.)  This 
could
be a simple abstraction to persist, retrieve, and delete saved configuration.

Original comment by Brett.Mi...@gmail.com on 8 Aug 2008 at 7:52

GoogleCodeExporter commented 8 years ago

Fixed in r1047 | Brett.Michael.Johnson | 2008-10-31 17:02:46 -0700 (Fri, 31 Oct 
2008) | 252 lines

This set of changes addresses Connector Manager Issue 78 -
Abstract Connector Configuration Storage.

One of the primary goals of this task is to move the various
bits of connector configuration out of the Java Preferences
(Registry on Windows, Preferences on Mac OS X and Linux, etc.)
Currently, the ConnectorState and ConnectorSchedule are stored
in the system or user's Preferences.

Another goal is to abstract the Connector Configuration Map
storage in a manner that is in line with the state and schedule
storage abstractions.  Currently, the outside world knows that
the Connector ConfigMap is stored as a Properties file.

A third goal is to push the knowledge of the storage mechanism
far down the food chain - as close to the connector instance 
as possible - so we aren't passing ConnectorStateStores and
ConnectorScheduleStores all over the place.

This is a partial implementation, with most of the core design
in place.  You should be able to see how your GData-based
implementation would fit into here (or, more importantly, how
this design can be altered to support a GData implementation).

The design as implemented so far:
 - A new ConnectorConfigStore interface mirrors the 
   ConnectorStateStore and ConnectorScheduleStore interfaces
   and stores/reads the connector configMap.

 - I enhanced PrefsStore, adding an implementation of
   ConnectorConfigStore.  I also reworked PrefsStore to
   lazily create Preferences nodes as required.  This avoids
   creating a bunch of empty Preferences nodes simply because
   a PrefsStore bean is created as a Legacy Store for upgrading
   old connector instances.

 - I implemented a FileStore implementation of the interfaces
   (similar to PrefsStore).

 - The ConnectorStateStore, ConnectorScheduleStore and
   ConnectorConfigStore implementations are still specified
   in the application context, but they are not initialized
   in setters to ProductionManager.  InstanceInfo pulls
   them from the application context using getRequiredBean().

 - Access to connector state, schedule, and config are
   through Instantiator Interface via getters and setters.
   These getters and setters are nearly all pass-through
   calls to InstanceInfo.

 - InstanceInfo knows about ConnectorScheduleStore,
   ConnectorStateStore, and ConnectorConfigStore. 
   It is also the highest level interface that knows 
   the ConfigMap is a Properties object.

 - The ConnectorConfigStore, ConnectorScheduleStore, and
   ConnectorStateStore interfaces now take a StoreContext
   object (rather than connectorName), that includes the
   connector name, connector directory, connector type name,
   [and in the future, other specialized information that
   might be need for a specific persistent storage
   implementation.]

 - Added MockConfigStore, similar to MockScheduleStore and
   MockStateStore.

 - Added LegacyConnectorScheduleStore, LegacyConnectorStateStore,
   and LegacyConnectorConfigStore configuration to the
   application context.  Attempt to upgrade existing
   connector instances by looking in the legacy stores if
   no data is stored in the configured stores.

 - Fixed up a huge number of tests that pass ConnectorStateStore,
   or ConnectorScheduleStore around.

 - Modifid the etc/applicationContext.xml in the source tree
   to reflect these changes.

 - Added FileStoreTest, similar to PrefsStoreTest.
   Enhanced PrefsStoreTest to test stored configurations.

 - The tests now compile and all pass.

Things I don't like about this implementation as it stands:

 - The Instantiator interface is the gateway to the stored
   connector state, config, and schedule via getters and setters
   that are simply pass-through calls to InstanceInfo.  Although
   the Instantiator is generally available to all those that
   need access to this information, it feels like these
   accessors are poorly placed.  For instance, retrieving
   the schedule in this snapshot looks like:
     instantiator.getConnectorSchedule(connectorName);
   whereas, I think something like this just feels "more right":
     instantiator.getConnectorInstance(connectorName).getSchedule();

   I guess I am not comfortable with the way InstanceMap,
   InstanceInfo, and ConnectorInterfaces relate to each other
   and the Instantiator and the outside world.  I would rework
   the Instantiator interface to consist of methods that only 
   create, destroy, and manage connector instances, and add a 
   ConnectorInstanceclass (or interface) that handles 
   interacting with individual connector instances.

 - The ConnectorConfigStore Interface deals with
   Properties objects.  Although InstanceInfo needs to
   deal with the Connector ConfigMap as a Properties
   object (for the Spring instantiation Resource and
   for PropertyPlaceHolderConfigurer), it did not need
   to expose the Properties aspect of the ConfigMap
   to the ConfigStore.  However, the load() and store()
   methods on Properties make a very convienient 
   implementation for the getConnectorConfiguration()
   and storeConnectorConfiguration() methods.
   The more abstract Map() interface lacks such easy
   serialization.

Change Log:
----------

M  
projects/connector-manager/source/java/com/google/enterprise/connector/instantia
tor/Instantiator.java
   - Getters and setters for connector config, state, and schedule.
   - For consistency, renamed dropConnector() to removeConnector().

M  projects/connector-
manager/source/java/com/google/enterprise/connector/instantiator/InstanceInfo.ja
va
   - Implementation of getters and setters for connector config, state, and schedule.
   - Retrieves ConnectorStateStore, ConnectorScheduleStore, and
     ConnectorConfigStore beans from application context.
   - Moved Property file management to PropertiesUtils.
     No longer exposed Properties file.
   - Replaced getProperties() with getConnectorConfig().
   - Removed internal Configuration subclasses introduced for
     ConnectorFactory change.  All instances are now effectively
     created using the MapConfiguration logic.
   - Construct InstanceInfo objects early.  It effectively
     replaces Configuration object for holding name, type,
     config map, etc during spring instantiation.
   - Create StoreContext for the Connector Instance persistent store needs.
   - Dropped different fromDirectory() and fromDirectoryAndThrow()
     implementations.  They all throw.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/instantiator/SpringInstantia
tor.java
   - Getters and setters for connector config, state, and schedule
     call through to InstanceInfo.
   - No longer holds or uses ConnectorStateStore.
   - For consistency, renamed dropConnector() to removeConnector().

M  projects/connector-
manager/source/java/com/google/enterprise/connector/instantiator/InstanceMap.jav
a
   - Handle changes to InstanceInfo interface.
   - No longer deals with Properties file.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/instantiator/EncryptedProper
tyPlaceholderConfigure
r.java
   - Move a couple of generic properties methods to PropertiesUtils.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/instantiator/ConnectorInterf
aces.java
   - No longer holds or uses ConnectorStateStore.
   - Dropped cached configMap, including in the constructors and
     getConfigMap() method.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/traversal/QueryTraverser.jav
a
   - No longer holds or uses ConnectorStateStore.
   - Uses InstanceInfo ConnectorState getters and setters.

A  
projects/connector-manager/source/java/com/google/enterprise/connector/common/Pr
opertiesUtils.java
   - Collect various Properties file management utility
     methods in one place.

A  projects/connector-
manager/source/java/com/google/enterprise/connector/common/PropertiesException.j
ava
   - Exceptions thrown by PropertiesUtils.

A  
projects/connector-manager/source/java/com/google/enterprise/connector/persist/S
toreContext.java
   - Storage Context object passed to Connector*Store interfaces contains
     connector name, connector directory, and [in the future] other context
     needed for future persistent storage implementations.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/persist/ConnectorScheduleSto
re.java
   - Add StoreContext parameter to methods for the benefit of
     FileStore implementation.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/persist/GenerationalStateSto
re.java
   - Add StoreContext parameter to methods for the benefit of
     FileStore implementation.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/persist/ConnectorStateStore.
java
   - Add StoreContext parameter to methods for the benefit of
     FileStore implementation.

M  
projects/connector-manager/source/java/com/google/enterprise/connector/persist/P
refsStore.java
   - Add StoreContext parameter to methods for the benefit of
     FileStore implementation.
   - Add implementation of ConnectorConfigStore for storing
     config map in preferences.

A  
projects/connector-manager/source/java/com/google/enterprise/connector/persist/F
ileStore.java
   - New ConnectorConfigStore, ConnectorStateStore, and
     ConnectorScheduleStore implementation that saves data
     in the Connector instance directory.

A  projects/connector-
manager/source/java/com/google/enterprise/connector/persist/ConnectorConfigStore
.java
   - Abstract interface for storing a connector config map,
     similar to ConnectorStateStore and ConnectorScheduleStore.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/manager/ProductionManager.ja
va
   - No longer holds or uses ConnectorScheduleStore.

M  
projects/connector-manager/source/java/com/google/enterprise/connector/manager/C
ontext.java
   - Make getRequiredBean() public so that InstanceInfo
     may access the Connector*Store beans.
   - Use PropertiesUtils to process the web app Properties file.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/scheduler/TraversalScheduler
.java
   - No longer holds or uses ConnectorScheduleStore.
   - Gets the connector's schedule via the Instantiator.

M  projects/connector-
manager/source/java/com/google/enterprise/connector/scheduler/HostLoadManager.ja
va
   - No longer holds or uses ConnectorScheduleStore.
   - Gets the connector's schedule via the Instantiator.

A  projects/connector-
manager/source/javatests/com/google/enterprise/connector/persist/MockConnectorCo
nfigStore.java
   - Mock ConnectorConfigStore, similar to MockConnectorStateStore, MockConnectorScheduleStore

M  projects/connector-manager/etc/applicationContext.xml
   - Switched ConnectorStateStore and ConnectorScheduleStore from PrefsStore to FileStore
   - Added LegacyConnector*Stores
   - Connector*Stores are no longer parameters to several other beans.

A  projects/connector-
manager/source/javatests/com/google/enterprise/connector/persist/FileStoreTest.j
ava
   - New test similar to PrefsStoreTest.

M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/instantiator/Instantiat
orTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/instantiator/InstanceIn
foTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/instantiator/InstanceMa
pTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/instantiator/MockInstan
tiator.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/instantiator/SetManager
ConfigTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/persist/GenerationalSta
teStoreTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/persist/MockConnectorSc
heduleStore.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/persist/MockConnectorSt
ateStore.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/persist/PrefsStoreTest.
java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/scheduler/HostLoadManag
erTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/scheduler/TraversalSche
dulerTest.java
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/traversal/QueryTraverse
rTest.java
M  
projects/connector-manager/testdata/contextTests/docPusher/applicationContext.xm
l
M  projects/connector-manager/testdata/mocktestdata/applicationContext.xml
M  
projects/connector-manager/testdata/contextTests/setManagerConfig/applicationCon
text.xml
M  projects/connector-manager/testdata/mocktestdata/MockRepositoryEventLog1.txt
M  projects/connector-
manager/source/javatests/com/google/enterprise/connector/test/ConnectorTestUtils
.java
M  
projects/connector-manager/testdata/contextTests/setManagerConfig/applicationCon
text.xml
M  projects/connector-manager/testdata/mocktestdata/MockRepositoryEventLog1.txt
   - Fix up tests to with regards to interface changes.

Original comment by Brett.Mi...@gmail.com on 7 Nov 2008 at 12:43

GoogleCodeExporter commented 8 years ago

Original comment by Brett.Mi...@gmail.com on 7 Nov 2008 at 12:45

Changed state: Fixed

GoogleCodeExporter commented 8 years ago

Original comment by jl1615@gmail.com on 12 Jan 2009 at 3:29

Added labels: Milestone-Release_1.3.0

GoogleCodeExporter commented 8 years ago

See also Issue 116 for additional work on this feature.

Original comment by Brett.Mi...@gmail.com on 10 Feb 2009 at 4:13

PILLUTLAAVINASH / google-enterprise-connector-manager

Fix config storage #78