Open smalers opened 2 years ago
smalers
commented
2 years ago From Doug Stenzel:
I do not believe we have a database datastore access to real-time data setup or is it necessary. We prefer that people either:
[No Action Required]
smalers
commented
2 years ago As for the State providing real-time data in a SQLite database that could be used to compare HydroBase with the telemetry station services, attached is a TSTool command file (with .txt added) that illustrates how a data dump could occur. This technique could also be used to dump self-contained datasets for modeling or other analysis. The reason that I suggest SQLite is that the resulting database would be one file that is easy to transport, is a "real" database that would allow queries, and could contain as much data as needed for test. I would figure out how to read the time series and compare to web service results.
An action item for me is to disable the streamflow real-time choices in TSTool and update documentation.
smalers
commented
2 years ago Based on Doug's comments, I have removed Irregular interval (which was used for real-time data) from the following data types in TSTool for the HydroBase database datastore and input type. The underlying code will still attempt to read but the UI does not offer the choice. If this works out OK then underlying code can be further refactored.
AdminFlow - AdminFlow (has multiple intervals)Hardware - Battery (real-time only)Climate - Precip (has multiple intervals)Climate - Temp (has real-time only)Reservoir - PoolElev (has real-time only)Reservoir - Release (has real-time only)Reservoir - Stage (has real-time only)Reservoir - Storage (has real-time only)Stream - WatTemp (has real-time only)Well - WellLevel (has multiple intervals but have been phasing out for years)Well - WellLevelElev (has real-time only)This resulted in some data types that only had real-time data being removed because no intervals remained. Note that the HydroBase database handling of real-time data is not as obvious as in web services, where "Telemetry Station" is used. HydroBase mixes the data types between structure types even if real-time data. All Irregular interval time series are now disabled in HydroBase datastore.
If I have done something wrong, code can be uncommented.
If further cleanup needs to be done, I will do that, but I may need State feedback. Keep this issue open for a while and allow State people to review.
These changes are included in TSTool 14.2.1.
In updating the TSTool documentation, I realized that TSTool is theoretically able to read real-time data from HydroBase, but I don't know if the feature works or is needed.
Real-time data can theoretically be accessed via "HydroBase" datastore (or other HydroBase database datastore), Streamflow, Irregular data type. This functionality was enabled many years ago when TSTool could access the database directly over the internet and because TSTool was used by State staff to view real-time data. Today, the public can certainly not access a database on the State server but someone inside the State's network could connect to the database directly if they had a database login and know how to configure a datastore. It may be that there are different databases in the State's system that need multiple datastores to be defined.
Lacking VPN access to the network, I have no idea whether this functionality still works, needs to work, etc. I would also have difficulty doing development, although I could do it if the State provided a downloadable database with real-time data. Now that web services are enabled, maybe they are relied upon for real-time data and no direct database connection is needed for real-time data.
On the other hand, why is a direct connection useful? Performance is faster with database connection. SQL could be used in the State's VPN if any HydroBase datastore is configured. It is also very useful for automated testing to be able to compare data from a database query compared to web service. I have found many issues with web services by doing comparisons. It would be possible to compare database values if the State did a dump to a form that TSTool could read, for example using TSTool SQLite database features.
Can the State team (DSS team?) evaluate whether database datastore access to real-time data is enabled, works, needs to be disabled, needs to be enhanced, etc.? This could avoid confusion for other users.