Add datastore to software design

[smalers]

It is becoming obvious that that some of the new advanced features require software components to manage data access, in particular for time series. Rather than reinvent the design, I recommend learning from the experience of TSTool, which uses datastores, as described below. Similar code can be implemented in TypeScript.

The core concept is that data access for any specific source occurs through a DataStore object. A datastore provides functionality to read from a specific format, database schema, or web service API. In Java code, there is a base DataStore interface and implemented classes for database and web services. Database datastores allow common features such as SQL integrations. Web service datastores typically use REST so there is opportunity to put some common features in the DataStore class. See the following. The package folders implemented in TypeScript should match the Open Water Foundation naming.

• datastore package
• DataStore interface
• WebServiceDataStore interface
• ColoradoHydroBaseRestDataStore - implementation of a web service data store for Colorado's HydroBase

Note that datastores do not only focus only on time series, although for the work we do being able to read time series is important and includes method(s) named readTImeSeries.

TSTool does not currently implement the concept of a FileDataStore but that is something that I have been envisioning to unify the TSTool interface, so that Input Type can be removed and just use a datastore. However, since the InfoMapper work is new, it may makes sense implement a DateValueDataStore class that implements FileDataStore behavior, which would allow a URL to be read, and would have a readTimeSeries method similar to now. The datastore class would just have knowledge of DateValueTS. A similar thing could be done for StateModTS. I could implement this in Java at some point, in which case a local file or URL could be read. One complication is that in a TSID, the part at the end such as ~HydroBaseWeb is used to look up the datastore for reads and this must be handled across all TSID.

Datastores for public web services could be defined in InfoMapper and created in lazy fashion. The following discussion focuses on how that might be managed.

It may make sense to introduce a concept of a a "Data Manager" similar to window manager and map layer manager. I hesitate to use the word "data" because it is so generic, and may it should be "DataStoreManager". This could contain a list of datastores that are global to the application. This object could be a singleton. The object could then have a method such as "readTimeSeries", similar to the following:

TSEngine - see the readTimeSeries0 funcation near line 5000.

The application code could then connect the various code to make things work. It may be necessary to abstract some things. For example, code that retrieves time series for visualization may need a way to delegate that work to the DataStoreManager, done in a general way. In Java this would be done with an interface.

[smalers]

I've been thinking about the file datastores here are some thoughts for use cases and requirements.

1. A "built in" datastore named StateMod could configured that would be used in a general way, as if a static function. For example, a TSID like 1234.xxx.Diversion.Month~StateMod~path/to/file would behave similar to before. The handling of path would need to be clearly documented, such as defaulting the root URL to that of the application configuration file or some other file (need to decide).
2. An application datastore named anything, for example StateMod1 would require a datastore configuration for the application, which would define a root URL under which paths are found. The TSID like 1234.xxx.Diversion.Month~StateMod1~path/to/file would result in a URL rootUrl/path/to/file.

Similarly, a datastore with generic name DateValue could be defined.

The application would have DataStoreManager instance (or similar) with a list of datastores, including the built-in datastores and any defined in the application configuration. The application configuration will be something like the following (see the TSTool datastore reference configuration files).

"datastores" : [
  {
     "name" : "StateMod",
     "type" : "owf.datastore.statemod",
     "rootUrl" : null
  },
  {
     "name" : "StateMod1",
     "type" : "owf.datastore.statemod",
     "rootUrl" : "https://...."
  }
]

The value of rootUrl could accept properties, like ${appDataPath} or accept https://... on same or different server.

The DataStoreManager instance would have a method to loop through the registered datastores and match the TSID datastore. Once matched, the appropriate "readTimeSeries" method would be called, using the appropriate class instance.

These ideas need to be expanded to define how to handle different situations, such as URL on different servers like GitHub.

[Nightsphere]

Basic Datastore functionality has been added, and documentation can be found under the
Appendix - Install and Config > app-config.json > Datastore (datastore) Properties
section in the InfoMapper User Documentation repo. Any other changes or updates can be opened as new issues similar to Issue #33.

Closing this issue.