search

USEPA / Stormwater-Management-Model

Dynamic hydrology-hydraulic water quality simulation model
236 stars 174 forks source link

NetCDF Output Format #41

Closed cbuahin closed 1 year ago

cbuahin commented 4 years ago

Would there be any interest in using the NetCDF as the output file format for SWMM? It is self documenting, portable, and is able to scale well.

dickinsonre commented 4 years ago

Of course, someone(s) will have to do this https://www.unidata.ucar.edu/software/netcdf/ test it, explain it, document it and make it backward compatible with the existing inp format. The advantage of the existing inp format is you can see it and save it in a form people in the future can use.

cbuahin commented 4 years ago

@dickinsonre, my proposal is only to write the output (i.e., .out) in NetCDF. The input file (i.e., .inp) will remain unchanged. I can implement it and share it for consideration. Would it be better to contribute it to OWA instead?

michaeltryby commented 4 years ago

Hey @cbuahin, This is a good a place as any to discuss things :smile:

NetCDF sounds promising! I had been toying with the idea of a HDF5 binary output writer: https://www.hdfgroup.org/solutions/hdf5/

I believe HDF5 is the underlying format used for pandas data frames. It would be pretty cool to generate a SWMM output file in a format directly accessible from pandas! It would give Python users complete unfettered access to their SWMM simulation results.

We could program this feature to be modular, so it could potentially support multiple output formats - classic, HDF5, NETCDF, ..., eventually something even better. Then the user could chose the best format for their particular application.

MattAndersonPE commented 4 years ago

XMDF as a solution for Extensile Model Data Format ( https://apps.dtic.mil/dtic/tr/fulltext/u2/a461368.pdf) should be considered. It uses HDF5 as the format and compression. https://en.wikipedia.org/wiki/XMDF#cite_note-1

On Tue, Apr 21, 2020 at 8:28 AM Michael Tryby notifications@github.com wrote:

Hey @cbuahin https://github.com/cbuahin, This is a good a place as any to discuss things 😄

NetCDF sounds promising! I had been toying with the idea of a HDF5 binary output writer: https://www.hdfgroup.org/solutions/hdf5/

I believe HDF5 is the underlying format used for pandas data frames. It would be pretty cool to generate a SWMM output file in a format directly accessible from pandas! It would give Python users complete unfettered access to their SWMM simulation results.

We could program this feature to be modular, so it could potentially support multiple output formats - classic, HDF5, NETCDF, ..., eventually something even better. Then the user could chose the best format for their particular application.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/USEPA/Stormwater-Management-Model/issues/41#issuecomment-617251860, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACDUFI5HSI7QW2CFO7R2ZCTRNW3RLANCNFSM4MNKXQFQ .

cbuahin commented 1 year ago

This will be integrated into future planning efforts.