search

OpenCDSS / cdss-app-statedmi-main

StateDMI main application, tests, and documentation
GNU General Public License v3.0
1 stars 0 forks source link

StateDMI - Network Builder Update? #75

Open MMITISEK opened 4 years ago

MMITISEK commented 4 years ago

There are multiple limitations to the existing StateMod Network builder. CDSS should consider an update to the software to utilize current platforms and technology. My primary issue with the network viewer is the limited extent of the viewer and the inability to print/process networks for use outside of the viewer. With increased detail and complexities required by today's models, the fixed extent requires continuous adjustment and scaling of nodes. The state should also consider the ability to georeference the network and/or background elements to provide a better understanding of a models/systems' spatial extent.

smalers commented 4 years ago

I'll provide a few comments for context. Specific feedback such as the printing limitations is helpful.

  1. Printing has always been a challenge due to variability of different print drivers, page sizes, etc.. I experimented a few years ago by adding a PrintNetwork command in StateDMI. Because I had a limited number of printers, I tried to test by writing to PDF (at the time had to pay for support) and Microsoft XPS format (built-in PDF-like format, but has limitations). In reality, it would be nice to print the network in large format, but few organizations have plotters. I added the command to automate tests for printing, and with the hope that it might be used to automate output for reports, presentations, websites, etc.. In the end, there was not funding or a project so I left it where it was. I think whatever approach is used, the ability to print or save to image all or part of the network would be helpful. It may be that the current software is close to meeting these requirements, but requires testing. It may be worthwhile to look at SVG as an output format. This was added to TSTool and StateDMI for graphing awhile back but needs revisited. Also, I will say that the CTRL-p (print) feature of many websites also has issues. Printing is not easy.
  2. StateDMI actually has some design features to deal with rescaling and offsetting the network to allow for growing the network. I believe that we did some work to help merge networks for the statewide dataset. Defining some specific use cases with corresponding tests and documentation could help. I generally recommend adding functionality in a command so the task can be automated and repeated. Maybe one or more commands is needed to manipulate the network. The network editor is used by a small number of people and sometimes workarounds such as editing the network file with a text editor was enough.
  3. The ability to georeference the network has been in the design from the start. The XML network file has two sets of coordinates. One is the page coordinate for diagram-based network, and the alternate coordinate is intended for use for geographic coordinates. The problem is that in the early days of putting networks together, modelers preferred diagrams, and we did not have good geographic coordinates in HydroBase. There are StateDMI commands to fill coordinates from HydroBase. However, the loose ends have not been dealt with. The StateMod GUI at one point had a "view diagram" and "view map" toggle that would switch the network display between the coordinate systems. HydroBase geographic coordinates are better now and the Source Water Route Framework would allow referencing StateMod data to the river network. Funding to revitalize the GUI was allocated this year but was put on hold given COVID-19 cuts. I don't know if we'll have to resubmit a proposal or if previous awards will be allocated at some point.
  4. As for "current platforms and technology", I'll need specific recommendations. The software is written in Java, which is one of the leading languages. We used XML for the network at the time because that was the leading technology for hierarchical data. More could be done to migrate to updated data formats such as using JSON. We had to custom-code the network editor because a solution was not available. My research on similar tools (RiverWare, MODSIM) showed that developers of those tools took a similar approach (build rather than buy).

Overall, the network editor could be improved and there are some loose ends that could be improved. The State will have to decide where to put its funding.

MMITISEK commented 4 years ago

Steve- I appreciate the quick reply.

Please let me know if there is anything more I can do to help.

-Mark

Mark Mitisek, PH Project Manager | Integrated Modeling Practice Lead Cell: 970-310-3835 Mark.Mitisek@LREwater.com http://www.lrewater.com/ LREWATER.COM http://lrewater.com/ 1221 Auraria Parkway, Denver, CO 80204

**CONFIDENTIALITY NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.

On Fri, Aug 28, 2020 at 1:35 PM Steve Malers notifications@github.com wrote:

I'll provide a few comments for context. Specific feedback such as the printing limitations is helpful.

  1. Printing has always been a challenge due to variability of different print drivers, page sizes, etc.. I experimented a few years ago by adding a PrintNetwork http://opencdss.state.co.us/statedmi/latest/doc-user/command-ref/PrintNetwork/PrintNetwork/ command in StateDMI. Because I had a limited number of printers, I tried to test by writing to PDF (at the time had to pay for support) and Microsoft XPS format (built-in PDF-like format, but has limitations). In reality, it would be nice to print the network in large format, but few organizations have plotters. I added the command to automate tests for printing, and with the hope that it might be used to automate output for reports, presentations, websites, etc.. In the end, there was not funding or a project so I left it where it was. I think whatever approach is used, the ability to print or save to image all or part of the network would be helpful. It may be that the current software is close to meeting these requirements, but requires testing. Also, I will say that the CTRL-p (print) feature of many websites also has issues. Printing is not easy.
  2. StateDMI actually has some design features to deal with rescaling and offsetting the network to allow for growing the network. I believe that we did some work to help merge networks for the statewide dataset. Defining some specific use cases with corresponding tests and documentation could help. I generally recommend adding functionality in a command so the task can be automated and repeated. Maybe one or more commands is needed to manipulate the network. The network editor is used by a small number of people and sometimes workarounds such as editing the network file with a text editor was enough.
  3. The ability to georeference the network has been in the design from the start. The XML network file has two sets of coordinates. One is the page coordinate for diagram-based network, and the alternate coordinate is intended for use for geographic coordinates. The problem is that in the early days of putting networks together, modelers preferred diagrams, and we did not have good geographic coordinates in HydroBase. There are StateDMI commands to fill coordinates from HydroBase. However, the loose ends have not been dealt with. The StateMod GUI at one point had a "view diagram" and "view map" toggle that would switch the network display between the coordinate systems. HydroBase geographic coordinates are better now and the Source Water Route Framework would allow referencing StateMod data to the river network. Funding to revitalize the GUI was allocated this year but was put on hold given COVID-19 cuts. I don't know if we'll have to resubmit a proposal or if previous awards will be allocated at some point.
  4. As for "current platforms and technology", I'll need specific recommendations. The software is written in Java, which is one of the leading languages. We used XML for the network at the time because that was the leading technology for hierarchical data. More could be done to migrate to updated data formats such as using JSON. We had to custom-code the network editor because a solution was not available. My research on similar tools (RiverWare, MODSIM) showed that developers of those tools took a similar approach (build rather than buy).

Overall, the network editor could be improved and there are some loose ends that could be improved. The State will have to decide where to put its funding.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenCDSS/cdss-app-statedmi-main/issues/75#issuecomment-683108793, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHU64DPPEJJFHZFH355Q7KTSDABIVANCNFSM4QONRUTQ .

smalers commented 4 years ago

Mark, you make a good point about coordinates. In the "old days", we were often printing the network on 11x17 and plotter sizes. At that time, people were working on CRT monitors and a 17" display was a major investment, so hard-copy output was almost a necessity. Now, with widescreen monitors and most work occurring electronically, there is much less need to accommodate hard copy. The reason for page layout was to reasonably scale to common page sizes so that when you printed you would get a good result.

So... if the State agrees that focusing on electronic form is OK, we can back off printer impacts on the design and make improvements. Printing will then have to live with the layout that the data defined. It would probably be relatively easy to write the network to an SVG file or other format that could be imported into other software.

As for importing data from other formats, including Excel, that would also be possible, if tributaries and confluences are properly handled. I already deal with generating a network in TSTool from a table, with the thought being that TSTool could implement point flow and other models that are driven mainly by time series.

We'll see what we can do when budgets recover from COVID-19.

smalers commented 4 years ago

Another topic that is related is the work on a new web-based StateMod dataset viewer. OWF has been working on this using a Severance Tax Grant and will be deploying a demonstration implementation soon. Currently the Upper Colorado Water Allocation Model website is hosted by OWF but will be moved to the OpenCDSS cloud storage bucket. This tool has the potential to provide a web-based, map-focused interface to StateMod datasets and help nudge desktop tools like StateDMI and StateMod GUI to more geographic displays. Most consumers of datasets may not actually need to run the model, which is why a web publication of the model is useful. In any case, there is a need for protocols to deal with locations that don't have identifiers or coordinates in HydroBase, which is being worked out.

MMITISEK commented 4 years ago

Very cool, thanks for sharing Steve!! I agree, it is a bit of a chicken and egg. Increasing the user base with a tool like this will hopefully encourage the State to support many of the desktop tools. I also think that it will push forward a map focused network and GIS data used in StateDMI or StateMod GUI. Please keep me posted.

-Mark

Mark Mitisek, PH Project Manager | Integrated Modeling Practice Lead Cell: 970-310-3835 Mark.Mitisek@LREwater.com http://www.lrewater.com/ LREWATER.COM http://lrewater.com/ 1221 Auraria Parkway, Denver, CO 80204

**CONFIDENTIALITY NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.

On Tue, Sep 8, 2020 at 4:31 PM Steve Malers notifications@github.com wrote:

Another topic that is related is the work on a new web-based dataset viewer. OWF has been working on this using a Severance Tax Grant and will be deploying a demonstration implementation soon. Currently the Upper Colorado Water Allocation Model http://opencdss.openwaterfoundation.org/datasets/colorado/2015 website is hosted by OWF but will be moved to the OpenCDSS cloud storage bucket. This tool has the potential to provide a web-based, map-focused interface to StateMod datasets and help nudge desktop tools like StateDMI and StateMod GUI to more geographic displays. Most consumers of datasets may not actually need to run the model, which is why a web publication of the model is useful. In any case, there is a need for protocols to deal with locations that don't have identifiers or coordinates in HydroBase, which is being worked out.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OpenCDSS/cdss-app-statedmi-main/issues/75#issuecomment-689170246, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHU64DPXCTGTWPKNX4NRWADSE2WFLANCNFSM4QONRUTQ .