Which is the current status of the tool?

[josiasritter]

Dear Developers,

I would like to use this tool to calibrate my Lisflood model setup for the Mekong Basin. Reading the information in the Readme and on the Lisflood homepage, it is unclear to me which is the current version of the tool and what are the requirements. When I clone the repository, I see that some work has been done to migrate the code to Python3, which is great! Is the migration close to being finished? Or will the development still take time and I should meanwhile use the original codes with Python 2.7?

A quick update on which is currently the best way to use the tool would be hugely appreciated.

Cheers and thanks for sharing this resource!

Josias

[doc78]

Dear Josias

Lisflood calibration tool is now compatible with python 3.7. However unfortunately the Readme file is outdated. The documentation will be updated as soon as possible, in the mean time if you have any question please write us and we will try to help to use the tool.

Cheers Carlo

[josiasritter]

Thanks for the quick reply, Carlo! Great to hear the tool is already fully available in Python 3.

Luckily the codes are quite well commented (thanks!) so I have been able to prepare all inputs and setting files and I advanced up to the start of the actual calibration (CAL_7_PERFORM_CAL.py). First, a couple of questions to make sure I have been on the right track up to here:

• CAL_4_EXTRACT_STATION.py: Do I assume correctly that this script is not needed if I want multiple stations to be included in the calibration?
• CAL_6_CUT_MAPS_list.py: Does this script cut only the static lisflood layers, or should also the climate forcings be placed into the input directory to be cut here, too?

When I now try to run the calibration (CAL_7_PERFORM_CAL.py), I immediately get the below error at the first calibration station, then the code advances to the next station and is stuck running without progress:

Calling "qsub -l nodes=1:ppn=32 -q long -N LF_cal_on_10501 /Users/ritterj1/PythonProjects/lisflood-calibration/scripts/runLF_on_10501.sh" sh: qsub: command not found

I understand that some software that includes qsub for queuing jobs is missing on my system. Do I need this software if I am running the calibration locally? And if so, how do I install it? I am running the calibration on a macOS Catalina with 8 cores.

Thanks a lot for your help!

Josias

[doc78]

Dear Josias

CAL_4_EXTRACT_STATION.py performs some checks and prepare (generates) the station_data.csv file and the observations file for each catchment in its dedicated "station" directory. Then you will need to call the the script CAL_7A_CALIBRATION.py and CAL_7B_LONGTERM_RUN.py on each single catchment, OR the script CAL_7_PERFORM_CAL.py on a list of catchments.

CAL_6_CUT_MAPS_list.py: you should include also the climate forcing into the same input directory to be cut.

"qsub" command is used to submit a job on our cluster of nodes, but you don't need it. You can just run the generated script "/Users/ritterj1/PythonProjects/lisflood-calibration/scripts/runLF_on_10501.sh"

Hope this helps Cheers Carlo

[EdgarEspitia]

Dear Carlo @doc78,

I am trying to use the calibration tool, but I got confused with the names of the python scripts on the documentation and release 0.2. After reading the comments above I understood that the readme is outdated, so I decide to follow the documentation with the python scrips, for example, using the script CAL_1_FILTER_STATIONS.py following the explanation of CAL_1_CAL_VAL_PERIODS.py in the documentation but I got some errors

CAL_1_FILTER_STATIONS.py [-h] settings_file stations_csv stations_type
CAL_1_FILTER_STATIONS.py: error: the following arguments are required: stations_csv, stations_type

I would appreciate some guidelines for calibration. Finally, I have some questions, what setting file .txt is required to follow a template? the LISFLOODSettings file should be the same to initialize LIFLOOD? the runLF_linux_cut.sh depends on the system we are running, so should I rewrite it? is it required to have installed LISFLOOD in the python virtual environment for calibration?

Thank you very much! Edgar

[doc78]

Dear Edgar

CAL_1_FILTER_STATION.py takes 3 arguments: 1) the settings file: you can find settings file template into the "integration" folder (setting.txt, settings_fast.txt and settings_slow.txt) 2) station_csv: this is the csv file "stations.csv" in "integration" folder. You can add the new station rows and the info required into the columns of the csv file 3) stations_type: this is the filter applied to the list of stations included into the stations.csv file. Only the station that have "station_type" value into the "EC_calib" column will be processed.

You can find in the "integration" folder some LisfloodSettings xml example files. The structure of the xml is the same used by Lisflood, so you can get info from the Lisflood 4.0.0 documentation (that is updated). This file will be used from the calib tool to tell the Deap Algorithm which modules should be used and the map paths (or fixed values) for the variables.

Yes, you need to install the lisflood package into your conda environment to use the calibration. You can install it using the command "pip install lislflood-model" or compile it from source code and then install it using the command "pip install ." in the lisflood-code folder

runLF_linux_cut.sh is an old script. You should instead use the CAL_4 and CAL_7 scripts in the actual master branch to execute the cutmaps and the calibration.

The readme will be updated as soon as possible with further details on all the new steps. In the meantime I hope this helps.

Best Regards Carlo

[EdgarEspitia]

Dear Carlo,

Thank you for your explanation, and quick reply.

I am still having one error KeyError: 'Spinup_days' in CAL_1_FILTER_STATIONS.py

How Spinup_days are defined? Is it defined in the station.csv file? And I supposed that CAL_TYPE and Min_calib_days will generate errors if they are not defined in station.csv file.

Best regards, Edgar

[doc78]

Yes, you are right, the station.csv file needs to be updated as well. You should add the Spinup_days column to the station.csv file as well as Min_calib_days. Furthermore, you should update the column "CAL_TYPE" with values "6" and "24" instead of the current "NRT_6h" and "HIST_24h" values. Spinup days are used to shift the calibration start date from to the forcing start date, as you see in liscal/station.py at line 78:

# A calibration requires a spinup
# first valid observation point will be at forcing start + spinup

Min_calib_day is used to exclude stations that do not have enough days of observations for the calibration.

Best Regards Carlo

[josiasritter]

Dear @EdgarEspitia,

By browsing through the code and looking at the inputs requirements of the scripts, I prepared my station.csv file as seen in the screenshot below. With this station.csv file, all the preprocessing scripts worked (up to including CAL_6_CUT_MAPS_list.py). Hope it helps!

Josias

[EdgarEspitia]

Dear @josiasritter and @doc78

Thank you very much for your help!

Now, in the script CAL_3_PREP_MISC.py, I got the error:

=================== START ===================
>> Reading stations data csv file...
>> Make map with station locations (outlet.map)...
col2map version: 4.3.3 (linux/x86_64)
nr. of records read: 1
nr. of records with mv value: 0
nr. of records with mv (x,y): 0
nr. of records outside map: 1
nr. of cells with mv: 34398
nr. of cells with more than one record: 0
nr. of cells with majority conflict: 0
>> Check for station conflicts...
pcrcalc version: 4.3.3 (linux/x86_64)
map2col version: 4.3.3 (linux/x86_64)
Station ID 1234 not found in outlet.map! Is there another station with the same location?
Number of station location conflicts: 1
Fix these! Enter 'c' to continue for now
Traceback (most recent call last):
  File "/my_system/lisflood-calibration/CAL_3_PREP_MISC.py", line 96, in <module>
    raise Exception("ERROR")
Exception: ERROR

I think the error is related to the coordinates of the outlet. Am I right? I would appreciate any suggestion.

Best regards, Edgar

[EdgarEspitia]

I am looking at what is wrong with the maps or coordinates. I think the error nr. of records outside map: 1 is related to the coordinates on ldd.map, so I checked the coordinates in the map and CSV files (all maps are in the WGS84 system), but all seem to be right. The only map that I suspect is the ldd.map because it was converted from the NetCDF to a .map format by using the command pcr2nc of the lisflood-utilities.

For using a nc2pcr is required 3 parameters nc2pcr -i /path/to/input/ldd.nc -o /path/to/output/ldd.map -c /path/to/clone.map -l, but I am not sure what is the clone.map so I did with out this parameter.

[doc78]

nc2pcr should work without clone.map in latest lisflood-utilities v0.12.19. (clone.map is only used to get coordinates to generate the ldd.map file in this case). However can you please check if ldd coordinates are correct? You can plot your ldd map together with tmp.map (in your temp folder) and gauges.map (in your output folder) and check which map is not correct. You should have ldd.map covering the station location, tmp.map having value 1 corresponding to the coordinates of the station and gauges.map having the ID value of the station at the correct station coordinates. You can also check if tmp.txt (in temp folder) is containing the correct coordinates and ID of the station.

[EdgarEspitia]

I found the source of errors, there were two mistakes, the first one was the coordinates, they were not the same in all CSV files, and the second was related to the station ID and Calibration ID, both of them were too long (more than 8 numeric characters).

After solving these errors, new ones arose:

?:1:84:ERROR: RUNTIME function accuflux: Unsound ldd ?:1:168:ERROR: /my_system/lisflood-calibration/tests/my_catchment/temp/accuflux.map: File '/my_system/lisflood-calibration/tests/my_catchment/temp/accuflux.map': No such file or directory

Thanks!

[doc78]

The new issue is related to the ldd map that is "unsound", that means not all the downstream paths end in a pit cell. You can fix it using the command "lddrepair" and generate the repaired map starting from your ldd map. You can find the documentation of the pcraster command here: https://pcraster.geo.uu.nl/pcraster/4.3.3/documentation/pcraster_manual/sphinx/op_lddrepair.html

[EdgarEspitia]

Thanks for your help! The command lddrepair worked well. I adjust the file names, and did it just by running the next script:

import pcraster as pcr
ldd = pcr.readmap("ldd.map")
result = pcr.lddrepair(ldd)
pcr.report(result,"ldd_corrected.map")

I continued the next steps, I ran CAL_6_CUT_MAPS_list.py just for one station, so I have a question, if I have multiple stations to calibrate, then I should add one column with the station and the observation, am I right?

And I supposed that CAL_6_CUT_MAPS.py do the same as CAL_6_CUT_MAPS_list.py , but just for the chose station, but I when I run it I got the error input mask file How can I check the outputs in this steep?

When I tried to run CAL_7A_CALIBRATION.py I got the error: ModuleNotFoundError: No module named 'lisf1', I solved it, by adding the path to the file lisf1.py in lisflood-code Finally, I got the error CAL_7A_CALIBRATION.py: error: unrecognized arguments: 2 when I run CAL_7A_CALIBRATION.py /my_system/settings.txt 633 1 2 As I understood, the parameters are the settings file, the ID of the stations' data to calibrate, the number of CPUs and a number to be used as a seed for a random number generator.

[doc78]

Dear Edgar

CAL_6_CUT_MAPS_list.py: yes, you should just add the list of the stations in one column in a text file and use the text filename as a parameter of the python script:

python CAL_6_CUT_MAPS_list.py

If you get the error "wrong input mask file", you should check the subcatchment_path in your settings_file and the mask.map file in your maps folder of the subcatchment_path. You need to provide a valid "mask.map" pcraster file to cut the maps.

CAL_7A_CALIBRATION.py: The seed is an optional parameter. You can just omit it or, if you want to use it, you should write the following command:

python CAL_7A_CALIBRATION.py /my_system/settings.txt 633 1 --seed=2

[EdgarEspitia]

I am still having problems, now related with the subcatchments. I got the next errors:

  File "/mysystem/lisflood-calibration/CAL_7A_CALIBRATION.py", line 66, in <module>
    subcatch = subcatchment.SubCatchment(cfg, obsid)
  File "/mysystem/lisflood-calibration/liscal/subcatchment.py", line 42, in __init__
    self.inflowflag, n_inflows = self.prepare_inflows(cfg)
  File "/mysystem/lisflood-calibration/liscal/subcatchment.py", line 81, in prepare_inflows
    upstream_catchments = [int(i) for i in stations_links.loc[self.obsid].values if not np.isnan(i)]

And I have some doubts

• how the subcatchments files (data and maps) should be arranged?

• what is the structure of station_links.cvs files?
  For example: Ai=428, Aj=380 ID,IDs of directly connected nested subcatchments,,,,,,,,,,,,,,,,,,, 380,,,,,,,,,,,,,,,,,,,, 428,380,,,,,,,,,,,,,,,,,,, Is it right?

Thanks for your help!

[doc78]

Dear Edgar. Can you please send us the full log of the error, since the error code is missing in your logs Thanks Carlo

[EdgarEspitia]

Dear Carlo,

I think the error is related to the paths in the settings file for the calibration or in the settings-Run.xml defining the outputs (lzavin.nc, and dis.tss). The error happens because the calibration algorithm cannot read the outputs of lisflood, so I think the problem is related to the structure of subcatchments (links and files structure ) and the paths in the settings files for calibration (for example: subcatchment_path, inlets, and interstation_regions). If you manually copy the outputs of the pre-run and run ( the files lzavin.nc, avgdis.nc, chanqWin.tss, and dis.tss) to the path reported in the error log, the calibration starts, but in the next iteration the error shows up.

error_log.txt

Thanks for your help!

[EdgarEspitia]

This is an example of the link structure

The link stations_links.csv looks like this:

6337350 6337340 6337610, 6337340, 6337350 6337330 6337320, 6337330, 6337610, 6337340, 6337350 6337310 6337512, 6337310 ,6337320, 6337330, 6337610, 6337340, 6337350 6337511, 6337512, 6337310 ,6337320, 6337330, 6337610, 6337340, 6337350

Is it right?

[doc78]

Dear Edgar. The stations_links.csv file seems OK to me.

I think the issue is in the xml file. You should check that pathOut and pathRoot are in the format:

<textvar name="PathOut" value="$(PathRoot)/out/%run_rand_id">

And variables are in the format:

<textvar name="DisTS" value="$(PathOut)/dis.tss">

The old style of these path are instead in the format:

<textvar name="PathOut" value="$(PathRoot)/out"> <textvar name="DisTS" value="$(PathOut)/dis%run_rand_id.tss">

And this format is not compatible with the previous. Please note that "%run_rand_id" will be replaced at runtime by the numeration of the generation and the run of the current step of the calibration, e.g. for generation "0" run nr 1 you will get "0_1", thus the output folder of this run will be:

/lisflood-calibration/tests/my_catchment/CATCHMENTS_DIR/633/out/0_1/"

If you still have the issue or have any doubt please send me your xml file and I will give it a look. Best Regards

Carlo

[EdgarEspitia]

Dear Carlo,

I am still having issues trying to run the script CAL_7A_CALIBRATION.py. I think, there is something wrong in the settings.txt in the section of path, exactly with: subcatchment_path, gauges_path, summary_path, interstation_regions, inlets

I would appreciate it if you have any comments on the attached setup.

Thanks for your help.

Best regards, Edgar error_log.txt settup.zip

[doc78]

Dear Edgar. The error you get is due to the wrong format of stations_links.csv. You have:

ID IDs of directly connected nested subcatchments
508
509
200,100,508,509

While it should be: ID, IDs of directly connected nested subcatchments,,
508,,,
509,,,
200,100,508,509

Please note that you should have at least 3 commas, and at least in the first row (the header), so that pandas will know that has to take 4 columns. Otherwise the pandas parser reading csv file will fail. The trick here is to add many commas on the first row, like the stations_links in the repository, to avoid issues if any catchment have a big number of inflows.

Best Regards Carlo

[EdgarEspitia]

Dear Carlo,

I fixed the stations_links.csv and ran all the calibration steps, but I am still having issues, look at the error log.

---------------------------------------
Gauge location 9.125 52.964
Upstream station(s):
Traceback (most recent call last):
  File "/my_system/lisflood-calibration/bin/CAL_7A_CALIBRATION.py", line 66, in <module>
    subcatch = subcatchment.SubCatchment(cfg, obsid)
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 42, in __init__
    self.inflowflag, n_inflows = self.prepare_inflows(cfg)
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 81, in prepare_inflows
    upstream_catchments = [int(i) for i in stations_links.loc[self.obsid].values if not np.isnan(i)]
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 81, in <listcomp>
    upstream_catchments = [int(i) for i in stations_links.loc[self.obsid].values if not np.isnan(i)]
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Thanks a lot!

Best regards, Edgar

[doc78]

Can you send me your updated csv file? Please note that catchments IDs should be just numbers, no letters nor special chars. Please check also for tabs or spaces.

[EdgarEspitia]

You are right, at the end of the lines in the file stations_links.csv was a tab, so I deleted it, and ran it again, but now the problems related to the subcatchments inflows. Look at the error log.

---------------------------------------
Gauge location 9.125 52.964
Upstream station(s):
Retrieving inflow for subcatchment 100
Traceback (most recent call last):
  File "/my_system/lisflood-calibration/bin/CAL_7A_CALIBRATION.py", line 66, in <module>
    subcatch = subcatchment.SubCatchment(cfg, obsid)
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 42, in __init__
    self.inflowflag, n_inflows = self.prepare_inflows(cfg)
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 93, in prepare_inflows
    raise Exception("ERROR: Missing " + Qsim_tss)
Exception: ERROR: Missing /my_system/lisflood-calibration/my_catchment/data/subcatchments/100/out/chanq_simulated_best.tss

[doc78]

Ok, this is fine. You just have to calibrate the catchment number "100" before the "200", since "200" need the "100" time series (as you wrote 200,100,508,509 in your stations_links.csv file). Once you calibrate the "100" you will get the chanq_simulated_best.tss file in the 100/out folder and so you will be able to go on with the calibration

[EdgarEspitia]

So that means we should follow the river network, in this case run the calibrations scripts for each station, at first the stations 509 or 509, second 100, and finally 200.

[doc78]

Yes, correct. There is also a script to run all catchments automatically in the correct order "CAL_7_PERFORM_CAL.py", that takes as input the full o a partial list of catchments, but it is build to submit jobs on our Cluster system, so you need to make some changes to use it on your system.

[EdgarEspitia]

I have tried , but I got an error. can be it related with the general inflow.map? what are the caracteristics of the general inflow.map?

---------------------------------------
Gauge location 9.72 51.004
Upstream station(s):
No upstream inflow needed

Found 0 inflows
Traceback (most recent call last):
  File "/my_system/lisflood-calibration/bin/CAL_7A_CALIBRATION.py", line 66, in <module>
    subcatch = subcatchment.SubCatchment(cfg, obsid)
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 44, in __init__
    self.resample_inflows(cfg)
  File "/my_system/lisflood-calibration/bin/liscal/subcatchment.py", line 59, in resample_inflows
    raise FileNotFoundError('inflow map missing: {}'.format(subcatchinlets_map))
FileNotFoundError: inflow map missing: /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/inflow/inflow.map

[doc78]

I think the general inflow.map is just missing. You should copy it to /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/inflow/inflow.map This is needed to generate a new file "inflow_cut.map" in the same folder, that will be cutted out from the general inflow map using the masksmall.map in maps folder. Additional info on inflow map are here: https://ec-jrc.github.io/lisflood-model/3_09_optLISFLOOD_inflow-hydrograph/

[EdgarEspitia]

It is strange this message, because the option of inflows in the xml is <setoption choice="0" name="inflow"/>

The /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/inflow/inflow.map is generated by the scripts, as well as the maps, to solve these issues I copied manually the masksmall.map, meteo, landuse, settings files (see attached figures).
is it possible to setup the directories from settings.txt?

Now lisflood runs but there some errors (see attached error_log.txt). # error_log.txt pirctures_files_structure.zip

[doc78]

This is still a work in progress and some cleaning up the the settings.txt and the scripts are still needed, that's why the generated folders are not corresponding right now to the requested ones. However you moved the files into the folders in the correct way, thus it should work now. Please check your XML file and your folders for the following paths: 'PrecipitationMaps', 'TavgMaps', 'ET0Maps', 'E0Maps'. It seems that at least one of this file is missing in your system. Please check also the PathMeteo variable in the xml file

[EdgarEspitia]

I managed to understand the required file structure to run the calibrations, but now I cannot figure out how to set up the paths and read the outputs for the next iteration.

<textvar name="PathOut" value="$(PathRoot)/out/%run_rand_id">
path:/my_system/lisflood-calibration/my_catchement/data/subcatchments/508/out/0_1/dis.tss

# error_log.txt files_structure.zip

[doc78]

Do you see any dis.tss or dis0_1.tss file in the output folder or in subfolders? Can you please send me the template file you are using as well as the files in the "settings" folder under your catchment? This seems a issue of the PreRun and Run setting files automatically generated, or an issue of the template file, with the rename of the %run_rand_id I explained previously, or the prerun failed. You should see also subfolders called 0_1, 0_2... in the "out" folder

[EdgarEspitia]

I did not find any dis.tss file in the output folder or subfolder related.

As a template, I am using the file settings_Run_xml.txt that was tested running Lisflood.

In the folder /my_system/lisflood-calibration/my_catchment/data/subcatchments/509/settings/ was generated the settings files: settings_RunPreRun0_1.xml
settings_RunRun0_1.xml settings_RunPreRun0.xml settings_RunRun0.xml settup_xml.zip

I think this is an issue renaming the output %run_rand_id as you explained previously.

[EdgarEspitia]

After cleaning all the folders for one defined subcatchment, running all the calibrations scripts, completing the required files, and finally running the script CAL_7A_CALIBRATION.py something is wrong, the folders 0, 0_1, ... are not created, the script just does not create the folders.

 ========================== LISFLOOD Warning =============================
Path defined in PathOut is not writable: /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/settings/out/0
  warnings.warn(LisfloodWarning(msg))

 ========================== LISFLOOD ERROR =============================
Missing files or misconfigured paths to run LisFlood, according activated modules. Please check your settings file /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/settings/settings_-Run_PreRun0.xml.
['\n\nPath defined in PathOut is not writable: /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/settings/out/0']

Otherwise, I created manually the folders 0, 0_1 and copied to them the lzavin.nc to force run the script, so I got the next error:

Discharge file path: /my_system/lisflood-calibration/my_catchment/data/subcatchments/508/out/0_1/dis.tss
Traceback (most recent call last):
  File "/my_system/lisflood-calibration/bin/CAL_7A_CALIBRATION.py", line 68, in <module>
    calibrate_subcatchment(cfg, obsid, subcatch)
  File "/my_system/lisflood-calibration/bin/CAL_7A_CALIBRATION.py", line 43, in calibrate_subcatchment
    calib_deap.run(subcatch.path, lock_mgr)
  File "/my_system/lisflood-calibration/bin/liscal/calibration.py", line 335, in run
    population = self.generate_population(halloffame)
  File "/my_system/lisflood-calibration/bin/liscal/calibration.py", line 277, in generate_population
    for ind, fit in zip(invalid_ind, fitnesses): # DD this updates the fitness (=KGE) for the individuals in the global pool of individuals which we just calculated. ind are
  File "/my_system/lisflood-calibration/bin/liscal/hydro_model.py", line 102, in run
    simulated_streamflow = self.objective.read_simulated_streamflow(run_id, self.cal_start, self.cal_end)
  File "/my_system/lisflood-calibration/bin/liscal/objective.py", line 71, in read_simulated_streamflow
    raise Exception("No simulated streamflow found. Probably LISFLOOD failed to start? Check the log files of the run!")
Exception: No simulated streamflow found. Probably LISFLOOD failed to start? Check the log files of the run!

[doc78]

Hi Edgar. First thing to check is the XML generated files: if the automatic settings XML files are generated correctly, you should see the correct path in the PathOut keyword. Once you are sure this is correct, the issue could be related to writing permission in the "out" folder. Please check if the script is able to write and create directories into the "out" folder.

[EdgarEspitia]

Hi Carlo,

I think the xml files generated and the template are right , please have a look at the attached files. settingsweser-Run.xml.txt settings_weser-Run_PreRun0.xml.txt settings_weser-Run_Run0.xml.txt

The output folder has read, write, and execute rights, so I can't figure out what is wrong.

[doc78]

Ok, probably that's because Lisflood itself is not creating automatically each output folder, but just writing output files in existing folders. Thanks for flagging this issue, I will see for possible solutions to be implemented in the calibration code or in directly in lisflood code. You can try it by creating the empty output folders and running again the calibration, just to check.

As a quick fix to this problem, please try to use the old output filename style, that is to use the template XML with this syntax:

<textvar name="PathOut" value="$(PathRoot)/out"> <textvar name="DisTS" value="$(PathOut)/dis%run_rand_id.tss">

You should use this syntax style for every output variable you use. Please have a look at the other template sample files in the "integration" folder like this one:

https://github.com/ec-jrc/lisflood-calibration/blob/master/integration/settings_lisflood_GloFASwuse_init_tss.xml

Pls let me know if this solves the issue.

[doc78]

Regarding the next issue, I think the xml is still incorrect for the initialization fields: you should use results of prerun (initialization run) as input for the lisflood run. To do that we will have to change also those names. When you are ready with the first test please send me the xml and I will check it further

[doc78]

Dear Edgar. After further investigation we found that the calibration tool should already create the output folders (they are created in hydro-model.py before starting each new generation and run). Thus I think the folder issue can be related to your liscal installation. Actually there is an important step that probably we missed to check: after downloading the source code you should install it typing "pip install ." If you forget this step, the CAL_XX scripts will still take the previously installed miniconda package of liscal. Thus I think you are still running an old version of the calibration tool. To be sure of that please type "pip uninstal liscal" in your conda environment, and then "pip install ." from the updated the lisflood-calibration source code folder.

[EdgarEspitia]

Dear Carlo,

I am running the scripts directly from the source code. I could not install liscal by using the pip tool.

[doc78]

Ok, let's try these steps to be sure of the version you are running (I do not have the issue on my side): 1) Please type "pip list" while in your conda environment and let me know if you find any "liscal" in the package list 2) open the hydro-model.py source script and check for the lines 60 and 93 that should have the makedirs line, creating your output directories

    os.makedirs(out_dir, exist_ok=True)

3) to be sure the path is correct, please add a print before the makedirs like this:

    print('creating out_dir ',out_dir)
    os.makedirs(out_dir, exist_ok=True)

and let me know what you get.

[EdgarEspitia]

There is no package liscal I added the line : print('creating out_dir ',out_dir)

It seems that the folder is created, but later when lisflood tried to read, it is searching in another folder.

error_log.txt

[doc78]

Ok, this is because you have a wrong PathRoot value. PathRoot should be the following:

<textvar name="PathRoot" value="%SubCatchmentPath">

[EdgarEspitia]

I changed the path. After that, It was necessary to copy all the maps and meteo to this subcatchment path, but now the lzavin.nc is required the lzavin.nc file, is it the output of pre_run and run lisflood?

[doc78]

I see in your setting file another issue. You have:

<setoption choice="0" name="InitLisflood"/>

while it should be:

<setoption choice="%InitLisflood" name="InitLisflood"/>

Because the calibration will generate initialization (prerun) settings files updating this variable. Please use this setting file to fix all the issue you have in yours:

https://github.com/ec-jrc/lisflood-calibration/blob/master/integration/settings_EFAS5.xml

This one is a working setting file, so you should keep all the variables and style of this one also in yours, just changing options, paths and filenames of your maps.

[EdgarEspitia]

Dear Carlo,

I fixed the issues with the variables names, so the calibration script works with the stations that are not nested. I tried to run for station 100 (see the figure ) but I got some errors (see the error log file).

River network

error_log.txt

[doc78]

Is this just because the row for the catchment '100' is missing in the stations_links file? It should be 100,508

[EdgarEspitia]

Thanks! It works fine.

Do you have any advice for choosing the parameters of the optimization algorithm?

For example:

[DEAP]
numcpus = NCPUS # 
min_gen =  5        # number of MIN generation to run  
max_gen = 16        # number of MAX generation to run  
mu = 18         # initial population  
lambda_ = 56        # size of generation of offsprings
pop = 15        # initial population
gen_offset = 3      # 
effmax_tol = 0.003  # 

# NOTES:
# numcpus   = ?  
# min_gen   = Run at least this number of generations
# max_gen   = Maximum number of generations to run, used as failsafe stop criterion. Takes precedence over minGen (handy for fast debugging)
# mu        = # of best children chosen to f    eed into the next generation (JRC decided to make it 2x calibration parameters)
#       This must always be at least 2, otherwhise DEAP cannot crossover-mutate the children
# Lamda     = # of children spawned at every generation (= # threads to run in parallel)
#       This must always be twice mu
# pop       = initial population
# gen_offset    = ?      
# effmax_tol    = ?