search

respec / HSPsquared

Hydrologic Simulation Program Python (HSPsquared)
GNU Affero General Public License v3.0
43 stars 17 forks source link

Reading Multiple WMD Files Containing Timeseries with Conflicting DSN Assignments #53

Open ptomasula opened 3 years ago

ptomasula commented 3 years ago

@PaulDudaRESPEC @TongZhai @aufdenkampe @bcous In trying to get Brendan's more complex WQ model to run we identified an issue with how UCI and WDM Readers handle multiple files that have timeseries with overlapping DSN values. This model contains 4 separate WDM files, some of which have conflicting DSNs between the timeseries. Presently the WDM reader overwrites timeseries with the same DSN with the most recently read timeseries.

HSPF appears to get around this issue by using the UCI file and its FILES specification. Later in the EXT SOURCES specification those file names (i.e. WDM1, WDM2) are used to distinguish between timeseries with the same DSN in different files. The UCI reader and WDM reader need to be expanded to capture and support this file naming.

It actually looks like this might have original been supported. Notably the parseD function in the ReadUCI reads the file name and returns it as part of the dictionary under the SVOL key. However it looks like line 416 then overrides that filename specification for an asterisk. I think removing that override should restore support when ReadUCI. Also the main.get_timeseries function already looks to have logic to read timeseries when the SVOL parameter is populated.

I propose the following as a path here to restore support for multiple WMDs:

  1. Remove the override of SVOL in ReadUCI
  2. Add logic in ReadUCI which also reads and writes the FILES specification to the ReadUCI
  3. Modify ReadWDM to read the file specification table, and if the input WDM filename matches a file in that specification write the timeseries to the SVOL/TS### key to be consistent with the get_timeseries function.

This approach will still allow users to read WDM files independently of the UCI file so we would retain the ability to read and view WMD files without running the model. However for model execution we'd now need to read UCI files first and then read WDM file(s). I'm curious if you all have any thoughts or alternative suggestions on how to best address these conflicting DSN values. If we are comfortable with the proposed solution I can start working on it in a feature branch.

PaulDudaRESPEC commented 3 years ago

Thanks for the thorough description. I've worked around this issue in the past by modifying the DSN numbering in the old HSPF model, and I think that's still a valid work-around if you need it immediately. Maybe the most complete solution would be to enhance the UCI reader to include an option to go ahead and also import the input EXT SOURCES from WDM to HDF5?

rburghol commented 2 years ago

Hey @ptomasula can you take a look at the reference issue and see if it fixes your issue? It is related in that it deals with multiple wdms needing their own specifier, thiugh don't know if it addresses the non-unique (to hsp2) DSNs.

https://github.com/respec/HSPsquared/issues/82