search

FLO-2DSoftware / qgis-flo-2d-plugin

A plugin for pre-processing/post-processing FLO-2D models
6 stars 7 forks source link

HDF5 output structure #971

Open FLO-2DKaren opened 1 year ago

FLO-2DKaren commented 1 year ago

Define the hdf5 file structure so that the hdf5 file is easy to review. This means grouping like data.

Grid Group GE x y n

Depth Group max depth final depth combined max depth combined final depth

Velocity Group

Mudflow group

Time group time to 2 ft time to 1ft time to max floodwave arrival time

Channel group

FLO-2D.zip

FLO-2DNoemi commented 1 year ago

OK, this is simple to implement.

On Wed, Jul 19, 2023 at 12:47 PM Karen @.***> wrote:

Define the hdf5 file structure so that the hdf5 file is easy to review. This means grouping like data.

Grid Group GE x y n

Depth Group max depth final depth combined max depth combined final depth

Velocity Group

Mudflow group

Time group time to 2 ft time to 1ft time to max floodwave arrival time

Channel group

FLO-2D.zip https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/files/12098141/FLO-2D.zip

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3UZRCX2O4IIPZ4PUKDXRAFS7ANCNFSM6AAAAAA2QGONXI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

FLO-2DKaren commented 1 year ago

@FLO-2DNoemi This message was for Robson. He's going to build a little processing script that sets up the structure and then give you the code so you can repeat the structure in flopro.exe. Don't do anything with hdf5 right now. Let's get the structure set up and then you can sort your code out. It will be much easier to work from a template so we don't have to do lots of runs.

FLO-2DNoemi commented 1 year ago

Perfect.

On Wed, Jul 19, 2023 at 1:49 PM Karen @.***> wrote:

@FLO-2DNoemi https://github.com/FLO-2DNoemi This message was for Robson. He's going to build a little processing script that sets up the structure and then give you the code so you can repeat the structure in flopro.exe. Don't do anything with hdf5 right now. Let's get the structure set up and then you can sort your code out. It will be much easier to work from a template so we don't have to do lots of runs.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1642504758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3UZUFBURILUDT5QRCLXRAMY7ANCNFSM6AAAAAA2QGONXI . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 1 year ago

Hi,

I'm working on this right now. Is there any documentation on the output tables? What each one of the tables mean. I'm a little confused in some of them.

rpachaly commented 1 year ago

I created a jupyter notebook for debugging while coding and a python script to change the hdf5 output structure. Here they are:

hdf5_struct_jupyter.zip

hdf5_out_script.zip

If you don't know how to use the jupyter notebook, just run the python script. It will ask for a hdf5 file using the easy gui library and then post process it. Be careful that I'm not creating a copy for the hdf5 and just updating it directly. The code is very straightforward and easy to adapt in other situations.

I'm not totally sure if I grouped correctly the datasets within the hd5f file, but it is easy to change the grouping for a more suitable review of the datasets. I created a folder named 'Other' for the datasets that I was not sure in which group to put in.

FLO-2DKaren commented 1 year ago

Hi Robson,

There sort of is. There's documentation for the ascii output. Part of your progress will be to help us with the documentation for these. Did you install FLO-2D? If so, you have the data input manual here: C:\Users\Public\Documents\FLO-2D PRO Documentation\flo_help\Manuals\

Chapter 5 of the data input manual has some structural info for the ascii output files.
We don't have the documentation for the hdf5 files so we'll need to build it. We can do a meeting and I'll help you with some basic information.

rpachaly commented 1 year ago

I'm available right now! Just send me the link

FLO-2DKaren commented 1 year ago

Just sent it.

rpachaly commented 1 year ago

FLO-2D_post.zip

FLO-2DKaren commented 1 year ago

@rpachaly Here's a start for group and dataset structure. I did a start on the attributes too. Maybe we can add a variable description to the attribute and that description includes the variable name.

Also let it be known that a lot of data is missing from that example. We will need to run more examples to get the full structure.

I believe the hdf5 groups and datasets are organized Alphabetically. I wonder if we can manipulate that or not?

Input
    Grid
        x (x coord of grid)
        y (y coord of grid)
        grid code (grid element number)
        elev (grid element elevation)
        manning (grid element manning's n value)
        neighbors (Grid element neighbors)
    Control
        cont (table of control variables)
        tol global (table of numerical stability variables)
Output
    Depth 
        Floodplain
            MaxDepth (grid element maximum depth including channel)
            MaxDepthFP (grid element maximum depth excluding channel)
            MaxWSE??? (grid element maximum water surface.  Which one is this???)
            MaxDepthTOL (max tol storage depending on the tol global or tol spatial)
            MAXWSELEV???  (One of these only has data for when the grid has water on it.  If the grid is dry, the data is missing.)  
    Depth 
        Channel
            DepCHFinal (Final cross sectional depth)
            MaxchDepth (Max cross sectional depth)
            MaxCHWSE (Max cross sectional WSE)
    Velocity
        Floodplain
            FinalVel
            MaxFPVeloc
            MaxVelDir
            VeltimeFP
            Vel_x_Depth
            VelDirec
            VelResMax
        Channel
            MaxCHVeloc
            VeltimeC
            VELCHFinal
    Time
        DefloodTime
        FloodTime
        FloodwaveArrivalTime
        TimeQMax8Directions
        TimeTo2Ft
        TimeTo1ft
        TimeToPeak
        Noemi, We need to add time to user defined depth.
        We also need depthdur 1 and 2 here. 
    Infiltration
        HydCond  This one looks more like hydraulic conductivity than a hydraulic structure.
        SoilDepth
        TotalInfDepth
    Hydraulic Structure
        HydStruct
        HydStructName...This one is still wrong in FLO-2D.  Robson, Figure out how to write a structure name to a single cell of a dataset.  The code that writes this is just writing the single digit of every name to a new row.
        HydStructQIN
        HydStructQOUT
        MAXQHYD
    Channel
        CHANMAX - this structure is a little wrong. Review the chanmax.out file.  We need to add the segment number and the channel element number to this table.
        CHANMAXNodes - this is just a table of nodes but I think it would be better if this was in the chanmax table.
        What do we do with HYCHAN.OUT???
rpachaly commented 1 year ago

Hi,

I was looking for the output variable definitions on the FLO 2D Data Input Manual. For the great majority of the variables I was able to figure out the description. I prepared a word document for us to keep track of these definitions: hdf5_output_structure.docx. There are some marked in red that I'm not totally sure if it is correct.

All these descriptions I'm adding to the hdf5 post processor script as attributes.

FLO-2DKaren commented 1 year ago

@rpachaly OK I'll update the document and reload it.

FLO-2DKaren commented 1 year ago

@FLO-2DNoemi

Noemi, can you do the ones I assigned to you. I'm not certain of some of these.

hdf5_output_structure.docx

We also need to get one with storm drain, one with two phase, and one with dam breach.

Noemi, can you please post the engine here so we can run a few more.

@rpachaly

Robson, you can run a few alternate models and get more data. This is good for you to learn since it will get you familiar with input data and output data in the ascii format. I'll post a few more here but you'll have to run them to get the modified hdf5 files.

FLO-2DKaren commented 1 year ago

Here are some runs you can do to get you different hdf5 results. https://flo-2d.sharefile.com/d-sc2ee81b9ef2b45e9a920d27312a3de1e

Noemi will send you the most uptodate engine and dlls.

Change the first line of cont.dat so that they are not silent and the hdf5 switch is on.

image

FLO-2DKaren commented 1 year ago

Here are the dlls. These go in the Project folder with flopro.exe and vc2005con.dll. https://flo-2d.sharefile.com/d-sbdac0f6c7106433b88cd4943dbdea8aa

double click flopro.exe to start a simulation.

rpachaly commented 1 year ago

Hi,

I'm running these projects but I'm getting only ASCII output files and not hdf5.

FLO-2DKaren commented 1 year ago
  1. you need the latestt exe from Noemi
  2. You need to turn on the switch in cont.dat @FLO-2DNoemi Noemi, can you help us with these hdf5 test runs?
rpachaly commented 1 year ago

I'm trying to figure out based on the ASCII out files. If the output variable name is slightly different, my code would be easy to adapt.

FLO-2DKaren commented 1 year ago

Yeah it will be easy to adapt. We want to use the other runs to identify missing datasets and grouping the mudflow and sediment transport results.

FLO-2DNoemi commented 1 year ago

Hi Robson,

Please download the latest exe from the following link:

https://flo-2d.sharefile.com/d-s5a78a14718d3475b85f6df664cde1558

You need to set up the IBACKUP as 3 in the CON.DAT. See below:

Line 1: SIMULT TOUT LGPLOT METRIC IBACKUP Example: 24.0 0.10 2 0 3

Please let me know if you have any questions or comments.

Noemi

On Thu, Jul 27, 2023 at 12:14 PM Karen @.***> wrote:

  1. you need the latestt exe from Noemi
  2. You need to turn on the switch in cont.dat @FLO-2DNoemi https://github.com/FLO-2DNoemi Noemi, can you help us with these hdf5 test runs?

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1653931043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3USMMREJVNSH63YY3LXSKHVHANCNFSM6AAAAAA2QGONXI . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DKaren commented 1 year ago

Thanks Noemi. I couldn't find my link.

FLO-2DNoemi commented 1 year ago

The following output files should be created if the component exists in the project. The only file that I have identified is not being written for all the projects with levee components is the LEVEE.OUT. I need to review this in the FLO-2D code.

The rest of the output files seem to be working fine according to my verification.

[image: image.png]

On Thu, Jul 27, 2023 at 1:26 PM Karen @.***> wrote:

Yeah it will be easy to adapt. We want to use the other runs to identify missing datasets and grouping the mudflow and sediment transport results.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1654072845, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3W3PFLLT3G4S6DWPIDXSKQEVANCNFSM6AAAAAA2QGONXI . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DKaren commented 1 year ago

Hi Noemi,

There's already some levee failure output data written to hdf5 too so maybe we can consolidate that.

rpachaly commented 1 year ago

Hi Robson, Please download the latest exe from the following link: https://flo-2d.sharefile.com/d-s5a78a14718d3475b85f6df664cde1558 You need to set up the IBACKUP as 3 in the CON.DAT. See below: Line 1: SIMULT TOUT LGPLOT METRIC IBACKUP Example: 24.0 0.10 2 0 3 Please let me know if you have any questions or comments. Noemi

Just ran a project here and I've got the hd5f file. Thanks Noemi!

FLO-2DNoemi commented 1 year ago

Good idea!

On Thu, Jul 27, 2023 at 1:46 PM Karen @.***> wrote:

Hi Noemi,

There's already some levee failure output data written to hdf5 too so maybe we can consolidate that.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1654105678, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3VBOOWU5OCPZPHQ5ILXSKSOZANCNFSM6AAAAAA2QGONXI . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DNoemi commented 1 year ago

Excellent, let me know if you have any questions or if you find something written wrong.

On Thu, Jul 27, 2023 at 1:49 PM Robson Pachaly @.***> wrote:

Hi Robson, Please download the latest exe from the following link: https://flo-2d.sharefile.com/d-s5a78a14718d3475b85f6df664cde1558 You need to set up the IBACKUP as 3 in the CON.DAT. See below: Line 1: SIMULT TOUT LGPLOT METRIC IBACKUP Example: 24.0 0.10 2 0 3 Please let me know if you have any questions or comments. Noemi

Just ran a project here and I've got the hd5f file. Thanks Noemi!

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1654112645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3TMOCKLZRP3CNJXGCLXSKS4FANCNFSM6AAAAAA2QGONXI . You are receiving this because you were mentioned.Message ID: @.***>

rpachaly commented 1 year ago

I created a google docs for the definitions and variables so we don't need to upload and download a word document. Here is the link: https://docs.google.com/document/d/1XpYevTJBDOuPx3f54kheA3guFYYre0F0b6hiPTN1BOY/edit?usp=sharing

I'm going to add the name of the new groups that'll be using on the hdf5 processor. If you don't like, just change in this google docs that I'll keep the same format on the python script.

FLO-2DKaren commented 1 year ago

You might have to add me and Noemi as contributors to the google doc. Our emails are gmail so it should be simple invite to edit on the document.

rpachaly commented 1 year ago

Just shared with both of you!

FLO-2DKaren commented 1 year ago

Excellent. I added some more notes to it.

rpachaly commented 1 year ago

Hi,

Would like me to describe the tables as well? I mean, what the rows and columns mean.

For some datasets are easy to figure out, based on the out files that have headers.

FLO-2DKaren commented 1 year ago

@rpachaly Yes please do that where ever you can.

rpachaly commented 1 year ago

@FLO-2DNoemi Hi,

I've finished the description of the hdf5 output file. Some comments about the datasets here:

The TIME group has many repeated datasets, isn't it a good idea to use only one dataset to represent the time steps?

I'm trying to figure out a way to avoid the alphabetical order of the groups/datasets, but it seems that the hdf5 file has to be created with the track order = True to create the groups/datasets in the order that we want. Otherwise, I'll have to create another hdf5 file and then save the groups/datasets. Is it possible to change this in the FLO-2D code?

I'm going to add the rows and columns description as attributes right bellow the dataset description on the hdf5 file. Something like this:

image

You can add your comments directly on the google docs (https://docs.google.com/document/d/1XpYevTJBDOuPx3f54kheA3guFYYre0F0b6hiPTN1BOY/edit#heading=h.52slwsugvh25)

As soon as I finish the rows & columns definitions, I'll send the code and the hdf5 output files here.

rpachaly commented 1 year ago

Hi,

Here is the hdf5 output processor: hdf5_out_processor.zip

Here are the output files processed for the projects that I've ran: Module 2 Bridge.zip Module 3 Breach.zip Module 5 Mudflow.zip Module 6 Breach Erosion.zip Module 7 Tailings Dam.zip Module 9 Storm Drain.zip

Let me know if everything is looking good. I've added the description, rows, and columns for each one of the datasets as attributes. Just a few are missing.

FLO-2DNoemi commented 1 year ago

Hi Robson,

Received. I will review the hdf5 output processor and the description and get back to you. I will try to do this today at the end of the day.

Thanks,

Noemi

On Mon, Jul 31, 2023 at 2:18 PM Robson Pachaly @.***> wrote:

@FLO-2DNoemi https://github.com/FLO-2DNoemi Hi,

I've finished the description of the hdf5 output file. Some comments about the datasets here:

  • ManholePop: I didn't find the definition for this dataset and I also couldn't figure out the rows and columns of the dataset.
  • MaxDepthTOL: I'm not totally sure about the description that I used.
  • VELTIMEFP: On the projects that I ran, I didn't find this dataset. The rows and columns descriptions are missing.
  • VelResMAX: The description says the maximum velocity is listed by the eight directions, but there are only three columns on the hdf5 output file.
  • TimeQMAX8Directions and TimeMAXQBYDIR: Aren't these two datasets the same?
  • HydStructTime: It starts at 0.1 and ends at 0 for the "Module 2 Bridge" project
  • OUTNQ_Time: The first two rows are 0 and it ends before the 16 for the "Module 9 Storm Drain" project
  • HydStructQIN: The results seems to be a little bit different from the .OUT files (see "Module 9 Storm Drain" project)
  • CHANMAX: The OUT file is separated by channels and the hdf5 is in sequence
  • OVERBANK: Results from OVERBANK.OUT and the HDF5 output file are slightly different (see "Module 2 Bridge" project). The node's order is inverted. I think that adding the node's name to the table would be nice.
  • XSECAREA: I'm not totally sure if the group that I placed this dataset is the best.
  • SDOUTFINGRID: I'm not totally sure if the row's description that I added is correct.

The TIME group has many repeated datasets, isn't it a good idea to use only one dataset to represent the time steps?

I'm trying to figure out a way to avoid the alphabetical order of the groups/datasets, but it seems that the hdf5 file has to be created with the track order = True to create the groups/datasets in the order that we want. Otherwise, I'll have to create another hdf5 file and then save the groups/datasets. Is it possible to change this in the FLO-2D code?

I'm going to add the rows and columns description as attributes right bellow the dataset description on the hdf5 file. Something like this:

[image: image] https://user-images.githubusercontent.com/39889306/257310142-3f9eade2-9b75-4c3e-ae18-11e2132f798b.png

You can add your comments directly on the google docs ( https://docs.google.com/document/d/1XpYevTJBDOuPx3f54kheA3guFYYre0F0b6hiPTN1BOY/edit#heading=h.52slwsugvh25 )

As soon as I finish the rows & columns definitions, I'll send the code and the hdf5 output files here.

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1658916412, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O3TZXK3GL5ODCZMY5OTXS7ZHPANCNFSM6AAAAAA2QGONXI . You are receiving this because you were mentioned.Message ID: @.***>

FLO-2DJJ commented 1 year ago

Hi Robson, I'm starting to get acquainted with the HDF5 functionality of the FLO-2D plugin. Could you allow me to see (share) the google HDF5 structure doc? Thanks.

rpachaly commented 1 year ago

Hi Robson, I'm starting to get acquainted with the HDF5 functionality of the FLO-2D plugin. Could you allow me to see (share) the google HDF5 structure doc? Thanks.

Hi JJ,

I shared with you last night. Let me know if you any problems seeing the document.

Thanks

FLO-2DJJ commented 1 year ago

Thanks Robson. I can see the document now.

On Mon, 21 Aug 2023, 1:03 pm Robson Pachaly, @.***> wrote:

Hi Robson, I'm starting to get acquainted with the HDF5 functionality of the FLO-2D plugin. Could you allow me to see (share) the google HDF5 structure doc? Thanks.

Hi JJ,

I shared with you last night. Let me know if you any problems seeing the document.

Thanks

— Reply to this email directly, view it on GitHub https://github.com/FLO-2DSoftware/qgis-flo-2d-plugin/issues/971#issuecomment-1686114263, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE32O73M52XB2SVQVL47DM3XWM56LANCNFSM6AAAAAA2QGONXI . You are receiving this because you commented.Message ID: @.***>

FLO-2DJJ commented 1 year ago

In relation to the question if the objects in a hdf5 file are accessed in any order (alphabetical or otherwise), Robson comment points in the right direction. They are accessed in ascending alphabetical order unless the group is created with track_order = True. At this moment I do not see whether that is important or not, what is relevant is the nice hierarchical structure. In any case, in Python the hdf5 file structure is represented by a dictionary metaphor and can be manipulated directly as such (recommended) or transformed by code as we wish.