lisflood files Vesdre subcatchment

[stevevdw28]

Good morning,

I have a problem with the area (mask) file. I have already used my area.nc file to run LISVAP and everything worked. Now that I want to use this file in LISFLOOD, there is an error telling me that I can only use PCRaster files. So I created a .map file. I debugged the problem but temporarily.... I now end up with this error:

File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/netcdf.py", line 34, in mask_array_np return data_cut[:, mask] IndexError: boolean index did not match indexed array along dimension 1; dimension is 1 but corresponding boolean dimension is 2740

If I understood correctly, this error occurs when splitting my meteorological variables on the basis of my input mask. I guess this is due to the fact that I am using a PCRaster file instead of a NetCDF file but I'm not sure about that.

I share with you via this sharing

(https://uclouvain-my.sharepoint.com/:f:/g/personal/steve_vandewijnckele_student_uclouvain_be/EtNc3rfH9m1OiFI5aASiJu8Bovm-j31GlZda8ouDeZo3mg?e=AvtwwL)

my files area.nc, area.map, my 5 weather variables as well as my file lisfloodSettings_reference.xml. I also made a screenshot of the error I get when typing the lisflood command lisfloodSettings_reference.xml.

Thank you in advance for your help...

[doc78]

Dear @stevevdw28

I have uploaded here a new netcdf file for the area mask, called "area2.nc". You can use this one to solve your issue. The issue is caused by a different order of the variables in the netCDF file and from a different origin point in the case of the pcraster map. The new version of lisflood will fix this issue.

Cheers Carlo

[stevevdw28]

ok great thank you but what is the code you used to reverse the dimensions of the nc file? I am having the same issue with my ldd.nc file. The structure of my ldd.nc file is exactly the same as your area2.nc file except that the order of the dimensions is reversed. So I would like to have y before x. I added the ldd.nc file to the shared onedrive. Thank you !!

[doc78]

Dear @stevevdw28 You can install the "development" branch of lisflood that already solves the problem (it will be released as version 4.1.3 next week), and use your original area.nc and ldd.nc maps. This version should work with all netCDF maps. Please let me know if this solves the issue Thanks Carlo

[stevevdw28]

Dear @doc78

Thank you for your help but I have just charged the files settings.py, netcdf.py and I have used the original area.nc but the problem is not resolved...

Thanks a lot

[doc78]

I think you need also "add1.py". I don't have your full setup but I tried using your original area.nc and ldd.nc maps as well as your meteo maps and it was working.

[stevevdw28]

Dear @doc78 ,

I reinstalled everything and still got the error with my starting area.nc file and ldd.nc..

Traceback (most recent call last): File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/global_modules/add1.py", line 196, in loadsetclone iterSetClonePCR(filename) File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/global_modules/zusatz.py", line 412, in iterSetClonePCR return remoteInputAccess(pcraster.setclone, file_path, error_msg) File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/global_modules/zusatz.py", line 430, in remoteInputAccess obj = function(str(file_path)) TypeError: Cannot open '/home/elic/stevevdw/Documents/lisflood-code/src/maps/mask/area.nc'. Note: only the PCRaster file format is supported as input argument.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/elic/stevevdw/.local/bin/lisflood", line 7, in exec(compile(f.read(), file, 'exec')) File "/home/elic/stevevdw/Documents/lisflood-code/bin/lisflood", line 41, in sys.exit(main()) File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/main.py", line 222, in main lisfloodexe(lissettings) File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/main.py", line 115, in lisfloodexe Lisflood = LisfloodModel() File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/Lisflood_initial.py", line 91, in init self.MaskMap = loadsetclone('MaskMap') File "/home/elic/stevevdw/Documents/lisflood-code/src/lisflood/global_modules/add1.py", line 208, in loadsetclone nr_rows, nr_cols = nf1.variables[value].shape # just use shape to know rows and cols... ValueError: not enough values to unpack (expected 2, got 1)

[doc78]

Dear @stevevdw28 Are you using the "development" branch of lisflood?

In the development branch line 208 and 209 of add1.py are the following:

        varname = [v for v in nf1.variables if len(nf1.variables[v].dimensions) == num_dims][0]
        nr_rows, nr_cols = nf1.variables[varname].shape  # just use shape to know rows and cols...

Please download the correct branch and try again. Thanks Carlo

[stevevdw28]

Dear @doc78

I use the development branch but the problem still persists..

[doc78]

Dear @stevevdw28 You are not using development branch yet. The code you are running is still the one in the package you installed, that is still the version 4.1.2 (you can it at the beginning of the execution) To use the development branch you downloaded, please uninstal lisflood (using "pip uninstall lisflood-model") and then install the develompment branch you downloaded from github (by typing "pip install ." from the lisflood-code folder). Now, try just running the command "lisflood" and you should see version 4.1.3 at the top of the execution: if so, you are using the right version. Cheers Carlo

[stevevdw28]

Dear @doc78

thank you for your information. Sorry for the inconvenience. I'll try :)

[stevevdw28]

Dear @doc78

Now the model is running thank you very much for that but I have an error again later in the process... I still allow myself to share this error with you on this forum. During the pcr -> np conversion, there is an incompatibility in the arguments for a map whose object in memory is 0x7f40736bd530. How to know the object in question that is causing the problem? The full error is:

Traceback (most recent call last): File "/home/elic/stevevdw/.local/bin/lisflood", line 7, in exec(compile(f.read(), file, 'exec')) File "/home/elic/stevevdw/lisflood-code/bin/lisflood", line 41, in sys.exit(main()) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 222, in main lisfloodexe(lissettings) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 115, in lisfloodexe Lisflood = LisfloodModel() File "/home/elic/stevevdw/lisflood-code/src/lisflood/Lisflood_initial.py", line 183, in init self.misc_module.initial() File "/home/elic/stevevdw/lisflood-code/src/lisflood/hydrological_modules/miscInitial.py", line 63, in initial self.var.PixelLength = compressArray(self.var.PixelLengthPcr) File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/add1.py", line 253, in compressArray mapnp = pcr2numpy(map,np.nan) File "/opt/easybuild/soft/2021b/software/PCRaster/4.3.3-GCCcore-11.2.0-Python-3.9.6/python/pcraster/numpy_operations.py", line 16, in pcr2numpy return _pcraster.pcr2numpy(_pcraster.clone(), map, mv) TypeError: pcr2numpy(): incompatible function arguments. The following argument types are supported:

1. (arg0: pcraster._pcraster.RasterSpace, arg1: pcraster._pcraster.Field, arg2: float) -> numpy.ndarray

Invoked with: <pcraster._pcraster.RasterSpace object at 0x7f40736bd530>, 0.0, nan

Thank you in advance for your help which has been very valuable to me so far !

PS : All of my files are inside the shared OneDrive

[doc78]

Dear @stevevdw28

This issue is caused by the option "gridSizeUserDefined" that you set to "1". When this option is set to "1", lisflood is expecting two maps in the variables "PixelLengthUser" and "PixelAreaUser", while I see that you just put a value of "0" that is not a map. For this reason lisflood stops with the error.

In your specific case, you don't need these maps since your maps are already in meters, so you should just set gridSizeUserDefined option to "0"

Cheers Carlo

[stevevdw28]

Dear @doc78

The problem is solved. But I have yet another problem now with reading my LAI files. The error is similar to the one obtained initially with the reader of the nc file.

Traceback (most recent call last): File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/add1.py", line 544, in loadLAI map = iterReadPCRasterMap(pcrvalue) File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/zusatz.py", line 407, in iterReadPCRasterMap return remoteInputAccess(pcraster.readmap, file_path, error_msg) File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/zusatz.py", line 430, in remoteInputAccess obj = function(str(file_path)) RuntimeError: Raster /home/elic/stevevdw/lisflood-code/src/lai/laio0000.001: can not be opened. Note: only the PCRaster file format is supported as input argument.

However, I have a double error associated:

Traceback (most recent call last): File "/home/elic/stevevdw/.local/bin/lisflood", line 7, in exec(compile(f.read(), file, 'exec')) File "/home/elic/stevevdw/lisflood-code/bin/lisflood", line 41, in sys.exit(main()) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 222, in main lisfloodexe(lissettings) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 115, in lisfloodexe Lisflood = LisfloodModel() File "/home/elic/stevevdw/lisflood-code/src/lisflood/Lisflood_initial.py", line 194, in init self.leafarea_module.initial() File "/home/elic/stevevdw/lisflood-code/src/lisflood/hydrological_modules/leafarea.py", line 63, in initial self.var.LAIX.loc[i,veg] = loadLAI(binding[map_name], LAINam, i) File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/add1.py", line 559, in loadLAI mapC = compressArray(mapnp, pcr=False, name=filename) File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/add1.py", line 256, in compressArray mapnp1 = np.ma.masked_array(map, maskinfo.info.mask) File "/opt/easybuild/soft/2021b/software/SciPy-bundle/2021.10-foss-2021b/lib/python3.9/site-packages/numpy/ma/core.py", line 2908, in new raise MaskError(msg % (nd, nm)) numpy.ma.core.MaskError: Mask and data not compatible: data size is 4540, mask size is 12439600.

This second error tells me that my LAI file has different x dimensions than my mask. However, this is not the case when I look at my files.

Thanks in advance again for your help

[stevevdw28]

Dear @doc78

I'm still trying to interpret this error. I don't know if this is due to the fact that I have null values for my laii.nc file (because there is no irrigation in my study area) or if this is due to the laiofday.txt text file. For this last file, I used the example of the Po watershed application case in Italy. I felt that I did not have to change the laiofday.txt file given the correspondence between my files and the files used for the application case.

Should I open another issue since this one (see the title) is resolved?

Thank you again for your help

[doc78]

Dear @stevevdw28 This is still the same issue: your LAI file has y variable in descending order. This cause a wrong cutmap operation in Lisflood. In your map files you had two issues (actually lisflood was not compatible with the file structure you used): one was the order of variables in some of the maps, now fixed in the version 4.1.3, and the other one is the reverted y order, i.e. y coordinates are saved in descending order: the latter will be fixed in the next release of lisflood. While waiting for the fix, the only thing you can try is to revert the y order in your maps. Otherwise please wait for the fix, that will be done in the next few days. Cheers Carlo

[stevevdw28]

Hello, thank you for your answer,

I changed the order of the y coordinates within my LAI files (in ascending order). You can see this using this code on my .nc files:

import xarray as xr dataset_inv = xr.open_dataset('path/to/lai_life.nc') print(dataset_inv['y']) dataset_inv.close()

So the order of the array is the same as for file area.nc

However, the error persists but the type of error has changed and corresponds to that:

Traceback (most recent call last): File "/home/elic/stevevdw/.local/bin/lisflood", line 7, in exec(compile(f.read(), file, 'exec')) File "/home/elic/stevevdw/lisflood-code/bin/lisflood", line 41, in sys.exit(main()) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 222, in main lisfloodexe(lissettings) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 115, in lisfloodexe Lisflood = LisfloodModel() File "/home/elic/stevevdw/lisflood-code/src/lisflood/Lisflood_initial.py", line 194, in init self.leafarea_module.initial() File "/home/elic/stevevdw/lisflood-code/src/lisflood/hydrological_modules/leafarea.py", line 63, in initial self.var.LAIX.loc[i,veg] = loadLAI(binding[map_name], LAINam, i) File "/opt/easybuild/soft/2021b/software/ELIC_Python/1-foss-2021b/lib/python3.9/site-packages/xarray/core/dataarray.py", line 208, in setitem self.data_array[pos_indexers] = value File "/opt/easybuild/soft/2021b/software/ELIC_Python/1-foss-2021b/lib/python3.9/site-packages/xarray/core/dataarray.py", line 758, in setitem self.variable[key] = value File "/opt/easybuild/soft/2021b/software/ELIC_Python/1-foss-2021b/lib/python3.9/site-packages/xarray/core/variable.py", line 856, in setitem indexable[index_tuple] = value File "/opt/easybuild/soft/2021b/software/ELIC_Python/1-foss-2021b/lib/python3.9/site-packages/xarray/core/indexing.py", line 1166, in setitem array[key] = value ValueError: could not broadcast input array from shape (7901988,) into shape (12439600,)

I don't understand what is causing this...

My files are still accessible via the following link: https://uclouvain-my.sharepoint.com/:f:/g/personal/steve_vandewijnckele_student_uclouvain_be/EtNc3rfH9m1OiFI5aASiJu8Bovm-j31GlZda8ouDeZo3mg?e=lTmQnB

Thanks again for your help to better understand my errors.

[doc78]

Dear @stevevdw28 I see this is a different issue: you have different valid pixels in LAI maps compared to mask map Cheers Carlo

[stevevdw28]

Dear @doc78 ,

I come back to you first of all to thank you for having referred me for my previous problem. Now I'm stuck just before the official initialization of the model for my watershed. I have this error which is displayed according to the following structure:

/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/add1.py:454: UserWarning: WARNING: valid_range not used since it cannot be safely cast to variable data type mapnp = nf1.variables[varname][cut2:cut3, cut0:cut1]

========================== LISFLOOD Simulation Information and Setting =================== ========== [X] LISFLOOD is used in the Deterministic [X] RUN INITIALIZATION [X] The simulation output as specified in the settings file can be found in /home/elic/stevevdw/lisflood-code/src/out [X] Activated modules: InitLisflood,SplitRouting [X] Report options: repDischargeTs,repEndMaps,reportedmaps

1 - 01/01/2021 00:00

ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/elic/stevevdw/.local/bin/lisflood", line 7, in exec(compile(f.read(), file, 'exec')) File "/home/elic/stevevdw/lisflood-code/bin/lisflood", line 41, in sys.exit(main()) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 222, in main lisfloodexe(lissettings) File "/home/elic/stevevdw/lisflood-code/src/lisflood/main.py", line 155, in lisfloodexe model_to_run.run() File "/home/elic/stevevdw/lisflood-code/src/lisflood/global_modules/zusatz.py", line 139, in run self._runDynamic() File "/opt/easybuild/soft/2021b/software/PCRaster/4.3.3-GCCcore-11.2.0-Python-3.9.6/python/pcraster/framework/frameworkBase.py", line 371, in _runDynamic self._userModel().dynamic() File "/home/elic/stevevdw/lisflood-code/src/lisflood/Lisflood_dynamic.py", line 116, in dynamic self.soilloop_module.dynamic_soil() File "/home/elic/stevevdw/lisflood-code/src/lisflood/hydrological_modules/soilloop.py", line 645, in dynamic_soil soilColumnsWaterBalance(self.index_landuse_all, self.is_irrigated, self.is_paddy_irrig, paddy_inactive, self.var.DtDay, SystemError: CPUDispatcher(<function soilColumnsWaterBalance at 0x7fbe776f0a60>) returned a result with an error set

This error does not specify which files are problematic. In view of the line of python code mentioned, I said to myself that this was due to the type of variable (float, int, byte, ...) considered. So I adapted my files like chan.nc, area.nc, lusemask.nc so that they only take two values 0 and 1 (byte). But nothing happens, the problem persists.

I would like to thank you again because thanks to you I am very close to obtaining a flow rate simulation which is an essential step in my work.

Thank you in advance for your help !

[StefaniaGrimaldi]

Dear @stevevdw28,

@doc78 and I looked at your problem together.

We tested the maps that you kindly shared with us here https://uclouvain-my.sharepoint.com/:f:/g/personal/steve_vandewijnckele_student_uclouvain_be/EtNc3rfH9m1OiFI5aASiJu8Bovm-j31GlZda8ouDeZo3mg?e=lTmQnB .

We do not know whether those maps are the most updated ones that you are currently using. Nevertheless, we were able to reproduce your error. This specific error is caused by errors in the maps of the soil properties. For instance, soil properties for the forest fraction (e.g. alpha1a_forest) have no-data/zero values in pixels for which the forest fraction is larger than 0. To avoid the error, you will need to correct the input maps of the soil properties.

We thought to share here a few links to the documentation about the static maps. The preparation of the static maps dataset for LISFLOOD is an extremely difficult task. Your work with 10m resolution maps is impressive, there are a few details to be corrected. We apologize in advance if some of the comments below are obvious, we are aware that the maps in the shared folder might be outdated.

https://ec-jrc.github.io/lisflood-code/4_Static-Maps-introduction/ All static input maps for LISFLOOD need to have the same model domain, resolution, and reference system.

https://ec-jrc.github.io/lisflood-code/4_Static-Maps_general-maps/ The mask map identifies the computational domain. All pixels with value 1 in the mask map will be used within the computations. All the pixels with values 1 in the mask map must be included (and have finite values) in all the LISLFOOD meteorological forcings and static maps. The mask map that is currently available in the shared folder has 1 in each pixel (the model computational domain is a rectangle): is it the last version?

https://ec-jrc.github.io/lisflood-code/4_Static-Maps_topography/ The LDD map is a critical input map. It shows the flow direction from each grid-cell to its downslope neighbor. In other words, the LDD map represents the path from upstream to downstream. This path must be continuous from the top of the mountains to the river mouth (or the outlet of your catchment). The LDD map which is currently provided in the shared folder has some problem because it does not allow the correct flow of water from upstream to downstream. More specifically, it seems to show features like highways and bridges: these features hinder the correct functioning of LISFLOOD.

https://ec-jrc.github.io/lisflood-code/4_Static-Maps_soil-hydraulic-properties/ This page lists the valid minimum and valid maximum values of each map representing soil properties values.

https://ec-jrc.github.io/lisflood-code/4_Static-Maps_land-use/ All the land cover maps are needed. You can have one or more landcover map with all zeros. For each pixel, the sum of the land cover fractions must be 1.

We hope that our answer allows you to make progress with the set-up of the model. Please let us know whether we can provide further information, Carlo and Stefania

[stevevdw28]

Good morning,

Thank you for your work and your precious help. You help me create, improve and match my files with the model.

I am adjusting the files related to the fraction of land occupation in order to see potential impacts of change on the outflows.

According to some configurations of my files, the model seems to work.

Cheers

Steve