HDF5 input structure - Githubissues

rpachaly commented 5 months ago


Input
    Grid
        x (x coord of grid)
        y (y coord of grid)
        grid code (grid element number)
        elev (grid element elevation)
        manning (grid element manning's n value)
        neighbors (Grid element neighbors)
    Control
        cont (table of control variables)
        tol global (table of numerical stability variables)
    Boundary Conditions
        inflow 
        outflow
    Infiltration
        infil
    Rainfall
        rain
        raincell
        flo2draincell
        raincellraw
    Evaporation
        evapor
    Storm Drain
        swmmflo
        swmmflort
        swmmoutf
        sdclogging
        swmmflodropbox
    Channels
        chan
        chanbank
        xsec
        chan_interior_nodes
        mult
        simple_mult
    Sediment
        sed
    Levee
        levee
        breach
    Tailings
        tailings
        tailings_cv
        tailings_stack_depth
    Hydraulic Structures
        hystruct
        bridge_xsec
    Streets
        street
        gutter
    ARF/WRF
        arf
    Buildings
        building_collapse
´´´

Still need to find a group
- Submerge_factor
- FPXSEC
- FPFROUDE
- TOLSPATIAL
- WSURF
- WSTIME
- TIMEDEPCELL
- SHALLOWN_SPATIAL
- OUTRC
- LID_VOLUME

rpachaly commented 5 months ago

@FLO-2DKaren @FLO-2DNoemi

I've been working on this simultaneously with the training. Here is the current state:

I'm exporting everything from QGIS into these folders and subfolders with attributes to explain the meaning of each table. This can be easily changed, but I tried to keep the exact same name and format as the *.DAT files. See the TOLER as example:

I think we should change some of the data structure on the HDF5 format, but my idea here is to start simple, close to what we already have on the *.DAT files, and then start changing the file's structure as we go.

There are still some data that needs to be exported, I'll be working on those. I'm really acquainted with the code right now.

Furthermore, I'm using compression and it is working REALLY well. I tested it on the Subdomain SD 3 project, the *.DAT files have around 187 mb and it goes down to 28 mb.

Let me know what you think and what can be improved. Don't need to hurry, this can be discussed after the training.

rpachaly commented 2 months ago

RAIN.DAT

FLO-2DNoemi commented 2 months ago

Robson, this looks good to me.

Let's do something similar with all the required data, then send it to me so I can read it with FORTRAN.

Remember to change the type of data from string to the corresponding type.

On Fri, Apr 12, 2024 at 10:28 AM Robson Pachaly @.***> wrote:

RAIN.DAT

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/dced2b8a-93de-4026-a816-122490c032f3

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2051868796, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3RJSTCPA2UTVT3QVALY47VPRAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRHA3DQNZZGY . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

CONT and TOLER -> They are ordered based on the Data Input Manual sequence that they appear on the CONT.DAT and TOLER.DAT files

Since we're using floats now, I set the null values to -9999.

rpachaly commented 2 months ago

GRID group

Maybe group X and Y?

FLO-2DNoemi commented 2 months ago

This looks good to me.

On Fri, Apr 12, 2024 at 1:11 PM Robson Pachaly @.***> wrote:

CONT and TOLER -> They are ordered based on the Data Input Manual sequence that they appear on the CONT.DAT and TOLER.DAT files

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/0112ec97-7855-421f-a583-19ec2e369c5e

Since we're using floats now, I set the null values to -9999.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2052154253, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3QF6VAPL4A25PJYGM3Y5AISRAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJSGE2TIMRVGM . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DNoemi commented 2 months ago

Yes, we can group X and Y.

On Fri, Apr 12, 2024 at 2:13 PM Robson Pachaly @.***> wrote:

GRID group

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/40df1a1d-fd11-4707-8396-c6e024478f3d

Maybe group X and Y?

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2052240526, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3WIIVHHXMORSBS3S6LY5AP2ZAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJSGI2DANJSGY . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

LEVEE -> This is the one that I did more changes on the structure, I created a specific dataset for levee failure and other for the levee data. However, I could not create a situation on the plugin where the fragility curve is used. @FLO-2DKaren, is the fragility curve for levees available on the plugin?

rpachaly commented 2 months ago

@FLO-2DNoemi Here is an hdf5 file with the already implemented datasets. When we have integers and floats on the same dataset, I'm setting the whole dataset to float. That's why in some cases the grid number is a float and in others it is an integer.

test.zip

FLO-2DNoemi commented 2 months ago

Robson,

Received. I will look into it and let you know my comments.

Thanks N

On Fri, Apr 12, 2024 at 4:38 PM Robson Pachaly @.***> wrote:

@FLO-2DNoemi https://github.com/FLO-2DNoemi Here is an hdf5 file with the already implemented datasets. When we have integers and floats on the same dataset, I'm setting the whole dataset to float. That's why in some cases the grid number is a float and in others it is an integer.

test.zip https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/files/14963469/test.zip

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2052496365, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3TVQUO62RIRUBTMC23Y5BA47AVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJSGQ4TMMZWGU . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DKaren commented 2 months ago

Hi Robson,

There are 3 components for the plugin that are not finished because no one uses them. Evaporation Streets Levee Fragility Curves

I'm not certain what to do about this. Noemi might have some ideas.

@FLO-2DNoemi @rpachaly

When you do levees, you might use a method like this where the 1 - 8 is crest elevation. It would be a float table. If the value is full that is the cutoff direction. If the value is NULL, it is no levee in that direction.

grid 1 2 3 4 5 6 7 8

I didn't look at your hdf5. So if you already did something else or better, cool.

rpachaly commented 2 months ago

SCS method:

Obs: INFIL_METHOD is always present if different than 0 Obs: INFIL_SCS_CELLS only appears if the infiltration was calculated for each cell. Otherwise, only INFIL_SCS_GLOBAL appears.

FLO-2DNoemi commented 2 months ago

This looks ok for infmethod equal to two.

I was able to read data from the HDF5 that you provided last Friday. The data looks good for CONTROL, GRID, RAINFALL and LEVEE.

Noemi

On Mon, Apr 15, 2024 at 8:53 AM Robson Pachaly @.***> wrote:

SCS method:

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/e2b16375-da0f-4d2f-82c3-9d263d69e18e

Obs: INFIL_METHOD is always present if different than 0 Obs: INFIL_SCS_CELLS only appears if the infiltration was calculated for each cell. Otherwise, only INFIL_SCS_GLOBAL appears.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2056783755, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3U7DSWK2OVI4VV4QXDY5PETXAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJWG44DGNZVGU . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

Perfect! I'm working on the infiltration.

rpachaly commented 2 months ago

HORTON:

Obs: INFIL_HORTON_CELLS only appears if the infiltration was calculated for each cell. Otherwise, only INFIL_HORTON_GLOBAL appears.

rpachaly commented 2 months ago

GREEN-AMPT:

Obs: INFIL_GA_CELLS only appears if the infiltration was calculated for each cell. Otherwise, only INFIL_GA_GLOBAL appears.

rpachaly commented 2 months ago

GREEN-AMPT and SCS:

It works exactly as the other infiltration methods.

rpachaly commented 2 months ago

Channel Infiltration:

These datasets are created only if INFCHAN = 1 on the INFIL_GA_GLOBAL.

Obs: I couldn't create the INFIL_CHAN_ELEMS using the plugin. I had to change the 'Channel Elements' table .

rpachaly commented 2 months ago

ARF:

FLO-2DNoemi commented 2 months ago

I do not remember getting a question or working on a project in the last few years that includes any of these.

I think we do not need to add 1D streets. I do not see the need to model streets with the 1D component in the future.

Evaporation is important to calculate for dispersion of pollutants so we should eventually add the feature to the plugin. Let's add this and the fragility curves to the github as an issue. Let's finish the first HDF5 and key developments for plugin 1.0, then we can add these two additional features.

On Sat, Apr 13, 2024 at 9:26 AM Karen @.***> wrote:

Hi Robson,

There are 3 components for the plugin that are not finished because no one uses them. Evaporation Streets Levee Fragility Curves

I'm not certain what to do about this. Noemi might have some ideas.

@FLO-2DNoemi https://github.com/FLO-2DNoemi @rpachaly https://github.com/rpachaly

When you do levees, you might use a method like this where the 1 - 8 is crest elevation. It would be a float table. If the value is full that is the cutoff direction. If the value is NULL, it is no levee in that direction.

grid 1 2 3 4 5 6 7 8

I didn't look at your hdf5. So if you already did something else or better, cool.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2053646367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3SRBB6UCBD5BDBPKYTY5EXBZAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJTGY2DMMZWG4 . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

For the OUTFLOW I did a video to explain how I organized the data. It is different, but I think it is better. Check the video and let me know what you think.

https://flo-2d.sharefile.com/d-sf1440a3701af431a99a4b54d96a7516d

rpachaly commented 2 months ago

INFLOW

The small caps are the ones that are different from the INFLOW.DAT

FLO-2DKaren commented 2 months ago

@FLO-2DNoemi @rpachaly

Hi Robson and Noemi, There is an error when you don't turn on the n value switch. FLOPRO won't run. So maybe we should just have that be on default.

rpachaly commented 2 months ago

HYSTRUCT

I combined rating curve and rating curve replacement into one table.

The BRIDGE_XSEC is on the Hydraulic Structure Group.

I combined the two Bridge lines (B) into one dataset.

rpachaly commented 2 months ago

FPXSEC

This one was a little tricky. I believe that this is the best format. The only issue is that, in order to keep all values as integers, I had to add the -9999 to the columns where no grid is present.

rpachaly commented 2 months ago

TOLSPATIAL

rpachaly commented 2 months ago

FPFROUDE

rpachaly commented 2 months ago

SHALLOWN_SPATIAL

rpachaly commented 2 months ago

GUTTER

rpachaly commented 2 months ago

MULT and SIMPLE_MULT

Hi @FLO-2DKaren and @FLO-2DNoemi, I was trying to export these files and I couldn't create them on the plugin. I was checking the db trigger and it is not coded yet.

rpachaly commented 2 months ago

MULT and SIMPLE_MULT

Hi @FLO-2DKaren and @FLO-2DNoemi, I was trying to export these files and I couldn't create them on the plugin. I was checking the db trigger and it is not coded yet.

This was a bug related to my geopackage. Everything is working.

FLO-2DKaren commented 2 months ago

Hi Robson,

I think the trigger used to be set to saving the user layers.

rpachaly commented 2 months ago

MULT and SIMPLE_MULT

I don't know what is the problem exactly, but sometimes adding the Multiple Channels and the Simple Multiple Channels works fine and sometimes it does not trigger the mult_cells/simple_mult_cells tables. However, the export to hdf5 is working.

FLO-2DJJ commented 2 months ago

I tested all the following situations without any problem:

Whenever a Multiple Channel Area is created in the User Layers (_multareas table) a trigger is fired to create Multiple Channel Cell features (_multcells table), one feature for each cell the area intersects with the grid layer.
Similarly, a trigger is fired when a Multiple Channel Line (_multlines table) is added, creating Multiple Channel Cells (_multcells table) from the intersection of the line with the cells of the grid layer.
Adding Simple Mult. Channel Lines (in _simple_multlines table) triggers the creation of Simple Mult. Chann. Cells features (in _simple_multcells table).
Deleting any of the above features of the User Layers causes the deletion of the cells intersected either by the area or the line.

rpachaly commented 2 months ago

Here is a video of the issue related to mult and simple_mult;

https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/dc203c12-15ba-446a-8f3c-90c0e10586c7

The geopackages: mult-bug-gpkgs.zip

rpachaly commented 2 months ago

CHAN, CHANBANK and XSEC

FLO-2DJJ commented 2 months ago

Hi Robson, Thank you for the video. I see the problem there.

Here is a video of the issue related to mult and simple_mult; mult.mp4

The geopackages: mult-bug-gpkgs.zip

I suspect the triggers, for some reason, are disabled in SelfHelpKit_v1.0.0.gpkg. I enabled them with self.gutils.enable_geom_triggers() and the problem was solved.

If that's the case, not only the multiple channel triggers were disabled but all the ones defined in the _"triggercontrol" table.

I included a enable_geom_triggers() in import_selected_components2 where it was missing (It will be merged into master in my next pull request from a branch I'm working for Storage Units).

There may be some others. I'm looking at them. Could you please check the use of disable_geom_triggers in the code you have been working with to see if there is any potential exception that could leave the triggers disabled? They look ok to me but I may miss the potential exceptions. Thanks.

rpachaly commented 2 months ago

I agree that the problem is related to the triggers, but I couldn't find where they are being disabled.

I will check for any disable_geom_triggers on the code that I'm working on.

Thanks for the fix JJ!

FLO-2DNoemi commented 2 months ago

Hi Robson,

This seems fine. I agree that it's the best format, especially considering that FP cross-sections have relatively few cells, resulting in a low number of no data entries.

Noemi

On Mon, Apr 22, 2024 at 10:24 AM Robson Pachaly @.***> wrote:

FPXSEC

This one was a little tricky. I believe that this is the best format. The only issue is that, in order to keep all values as integers, I had to add the -9999 to the columns where no grid is present.

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/4e1cf47f-e1e9-491b-8190-b3f839d77165

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2069641699, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3XEYD3TQM6SIDYVC3DY6UMQZAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRZGY2DCNRZHE . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

STORM DRAIN: SWMMFLO, SWMMFLORT, SWMMOUTF, SDCLOGGIING, SWMMFLODROPBOX

I did a bunch of modifications for this one and, in this case, I'm not totally sure if they are good or not haha.

I added the column Node Identifier/Outfall Identifier as a key to easily make relations between the tables and to avoid making string tables. The Node Identifier identifies all SD Inlets in the system. Outfall Identifier identifies all SD Outfalls in the system. I prefer this way instead of using the grid or the name.

Feel free to make suggestions.

FLO-2DNoemi commented 2 months ago

Hi Robson,

I reviewed this and it looks good. I do not see any issue. I will let you know if something arises once I read the HDF5 file.

Thanks, N

On Thu, Apr 25, 2024 at 5:25 PM Robson Pachaly @.***> wrote:

STORM DRAIN: SWMMFLO, SWMMFLORT, SWMMOUTF, SDCLOGGIING, SWMMFLODROPBOX

I did a bunch of modifications for this one and, in this case, I'm not totally sure if they are good or not haha.

I added the column Node Identifier/Outfall Identifier as a key to easily make relations between the tables and to avoid making string tables. The Node Identifier identifies all SD Inlets in the system. Outfall Identifier identifies all SD Outfalls in the system. I prefer this way instead of using the grid or the name.

Feel free to make suggestions.

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/f72fb292-5214-4475-82de-f51d97588baf

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/69ede58d-02a1-48a7-ba8c-cfc38465c410

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/7280aacd-18c2-4b4b-8c4b-f6d5d558b6cd

image.png (view on web) https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/78048f3f-fb64-4d35-a4c4-3fff1aaa6e6a

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2078201711, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3ROWYR33PMIQM3JFFTY7FYGLAVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYGIYDCNZRGE . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

Check this out @FLO-2DNoemi

https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/assets/39889306/495250d3-6762-4ace-8757-e99ace5112aa

rpachaly commented 2 months ago

To retrieve the INP in python:

with h5py.File(hdf5_file_path, 'r') as hdf:
    swmm_input_contents = hdf['SWMM_INP'][()]
    swmm_input_contents = swmm_input_contents.decode('utf-8')

    with open(output_inp_file_path, 'w') as output_file:
        output_file.write(swmm_input_contents)

FLO-2DNoemi commented 2 months ago

Hi Robson,

I like the idea of keeping the INI and INP files in the HDF5 as you showed in the video.

The FLOPRO will still read it from INP for this upcoming release.

Noemi

On Fri, Apr 26, 2024 at 10:48 AM Robson Pachaly @.***> wrote:

To retrieve the INP in python:

with h5py.File(hdf5_file_path, 'r') as hdf: swmm_input_contents = hdf['SWMM_INP'][()] swmm_input_contents = swmm_input_contents.decode('utf-8')
with open(output_inp_file_path, 'w') as output_file:
    output_file.write(swmm_input_contents)
— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2079542629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3UPCZFFRVYPJSSHCQ3Y7JSJ7AVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZZGU2DENRSHE . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DKaren commented 2 months ago

@FLO-2DNoemi @rpachaly

QGIS H5PY module updated with QGIS Version 3.36.
I don't think we will have too many problems with h5py module update or an hdf5 version update because they seem to be more compatible these days.

FLO-2DNoemi commented 2 months ago

Robson,

Please try loading the INP file, the one created from the FLO-2D HDF5 Data file with the additional line at the end of the INP file, into EPA SWMM GUI and see if it works.

Thanks,

Noemi

On Fri, Apr 26, 2024 at 11:08 AM Noemi Gonzalez @.***> wrote:

Hi Robson,

I like the idea of keeping the INI and INP files in the HDF5 as you showed in the video.

The FLOPRO will still read it from INP for this upcoming release.

Noemi

On Fri, Apr 26, 2024 at 10:48 AM Robson Pachaly @.***> wrote:
To retrieve the INP in python:

with h5py.File(hdf5_file_path, 'r') as hdf: swmm_input_contents = hdf['SWMM_INP'][()] swmm_input_contents = swmm_input_contents.decode('utf-8')
with open(output_inp_file_path, 'w') as output_file:
    output_file.write(swmm_input_contents)
— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/1158#issuecomment-2079542629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3UPCZFFRVYPJSSHCQ3Y7JSJ7AVCNFSM6AAAAABCFU42FKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZZGU2DENRSHE . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 2 months ago

QGIS H5PY module updated with QGIS Version 3.36. I don't think we will have too many problems with h5py module update or an hdf5 version update because they seem to be more compatible these days.

The h5py version that I'm using have a DeprecationWarning. I hope they had fixed it.

Please try loading the INP file, the one created from the FLO-2D HDF5 Data file with the additional line at the end of the INP file, into EPA SWMM GUI and see if it works.

It works. When I retrieve the INP file using python (h5py), the two files are identical.

FLO-2DKaren commented 2 months ago

Hi Robson,

I see you merged the code so I can test this this weekend.

It has been my experience that those hard returns are added when the file is loaded. I'll check it when I'm doing testing and storm drain documentation this weekend.

rpachaly commented 2 months ago

Yes! Everything that is commented here is implemented. Still need to export:

dam break or breach files
mud files
tailing files
building collapse

rpachaly commented 3 weeks ago

See https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/997#issuecomment-2171122814

FLO-2DSoftware / qgis-flo-2d-plugin

HDF5 input structure #1158