[hydro-as-2d] driver support for the xmdf result files

[dimdle]

Hi! We're using mdal/crayfish quite often, especially to show and analyse results of the 2d hydraulic model hydro-as-2d / hydro-ft-2d. ...and it would be great, if we can show and analyse the result from .h5 files in QGIS :)

Is a driver for the xmdf (.h5) format planned to get into mdal? maybe its just an extension to the already existing xmdf driver. hydro-as stores a relatively simple xmdf format with one group per dataset. Each group stores 4 tables (Mins, Maxs, Times, Values) I'm not into cpp, but I wrote a simple function in python using the h5py lib (see attached) to read/write the data. May this could help a little bit to get the driver for reading these files into mdal :)

hydras_h5_io.py.zip

test files testdata_hydro_as.zip

it would be great to see the hydro-as xmdf support in the future!! cheers

[PeterPetrik]

Hi @dimdle , we would need a test case with size < 3MB unpacked. XMDF is already supported I think, see https://github.com/lutraconsulting/MDAL/blob/master/mdal/frmts/mdal_xmdf.hpp ..?

[dimdle]

Hi @PeterPetrik thx for quick reply. the existing xmdf driver reads TUFLOW maybe BASEMENT datasets. Hydro-as is storing the results in other groups in the mxdf file. I'll generate a small example for testing tonight.

if it helps a can write a short description how the hydro-as data is stored in the xmdf files.

[PeterPetrik]

that would be good thanks.

[dimdle]

here's the testdata from a small model incl. .2dm mesh for_test_mdal.zip

[dimdle]

and here's a short description of the format (like in the mdal_xmdf.hpp):

/**
* Structure of the hydro-as / Aquaveo SMS xmdf file: 
*
* basically all datasets are stored in groups
* each dataset/group has got 4 Arrays: Mins, Maxs, Times, Values
* 
* - root
*     - File Type: xmdf (str)
*     - File Version: 2.1 (array)
*     - dataset1 (group)
*         - Maxs   : 1d Array storing Maximum value of all TS for each node in 2dm 
*         - Mins   : 1d Array storing Minimum value of all TS for each node in 2dm 
*         - Times  : 1d Array storing all output timesteps 
*         - Values : 2d Array with value of dset (one column per timesteps)
*     - dataset2 (group)
*         - Mins   : 1d Array ...
*         - Maxs ...
*           - ..
*           - ..
*      - dataset3 (group)
*         - ...
*/

[PeterPetrik]

hi @dimdle, we have fixed the issue and it will be part of the qgis 3.18 release and nightly builds sometime later this month. if you want to test beforhead, download MDAL from branch https://github.com/lutraconsulting/MDAL/pull/317, compile yourself, and try mdalinfo on your files to see if they are read correctly.

[dimdle]

hi @PeterPetrik, thank you guys for adding the new feature so fast!! I really like to test it with some other Datasets, but i got stock in compiling the branch :( i code most of the time in python and sql, but have no experience in compiling with cmake. sometime we just have to recompile old fortan code :) Is it possible to pass me a compiled version or link me a good tutorial how to compile such a project?

[PeterPetrik]

@dimdle what is your OS? it is merged , so you can compile from master now.you can check https://github.com/lutraconsulting/MDAL#windows or wait month to get to nightlies

[dimdle]

@PeterPetrik i'm on windows10 64 (with wsl2). yesterday i tried it in the wsl, (merged 317 to master locally) but probably I missed a step which is not in the description (or is to obvious to describe :) ).

[runette]

There is a trick that would probably work (no guarantees) using the Conda Build script. Assuming that you have conda loaded and running on your machine.

1 Clone the mdal-feedstock https://github.com/conda-forge/mdal-feedstock repo onto your local disk 2 cd to the mdal-feedstock folder in terminal 3 run :

conda create --name mdal // the name is not important as long as it is not in use already conda activate mdal

4 At this point if you run conda build recipe it would build version 0.7.2 5 However - you can go to recipe\meta.yml and change the repo used for the build.

I think that the lines :

url: https://github.com/lutraconsulting/MDAL/archive/{{ version }}.tar.gz sha256: 37c050ece0ad2cf0e04f8519f23a4125ab7f88f04f4989de8a2a5daaef846e02

should be replaced with :

git_url: https://github.com/lutraconsulting/MDAL.git

6 If you now run conda-build - it should build the new version. There should be a message at the end that tells you where the new files are

conda build recipe

(Note - do this from the root of the repo and not from the directory containing meta.yml

Paul

PS. - obviously - don't try to commit any changes back to mdal-feedstock.

On Thu, 12 Nov 2020 at 11:50, dimdle notifications@github.com wrote:

@PeterPetrik https://github.com/PeterPetrik i'm on windows10 64 (with wsl2). yesterday i tried it in the wsl, (merged 317 to master locally) but probably I missed a step which is not in the description (or is to obvious to describe :) ).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lutraconsulting/MDAL/issues/313#issuecomment-726030478, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARC2M34BFJTRFRMP4Z3WIDSPPDYFANCNFSM4TDXWLLA .