Support to test "LISVAP" from Linux terminal

[markdurel]

Hi community. It's about Lisvap. Indeed I cloned the repository from github on the Linux terminal. It worked well. I am unable to adjust the tests_efas.xml files to run the test. Also when I run the command: $ python lisvap... /../../tests_efas.xml -v , I get an error named: ModuleNotFoundError: No module named 'pcraster'. Screenshots are on the links below. https://twitter.com/mak_durel/status/1516873619846176773/photo/1 https://twitter.com/mak_durel/status/1516873619846176773/photo/2

[gnrgomes]

Dear Mark, Did you follow all the instructions to install PCRaster? https://ec-jrc.github.io/lisflood-lisvap/3_LISVAP_installation/

First try the instructions my colleague Stefania gave you by email: On your conda environment, if the name of your environment is "NAME_OF_YOUR_ENVIRONMENT", then I suggest trying the following commands: conda activate NAME_OF_YOUR_ENVIRONMENT conda install -c conda-forge pcraster gdal

[EdgarEspitia]

Dear @gnrgomes

I have some troubles running LISVAP. I would appreciate your help.

I have installed LISVAP in my system, it seems to be fine, but when I try to run any of the use cases (for example tests_efas.xml) I got an error, because it is not possible to read the base map area.nc.

Trying to access file /my_system/hydro_modelling/lisflood-lisvap/basemaps/area.nc: attempt n. 1 ... You can find the error output here lisvap_log.txt

I installed LISVAP by following two ways of three described in the documentation [1]. The Option 2. Install LISVAP python module using pip tool and Option 3. Clone the repository from the GitHub repository, but I got the same error reading the base map area.nc

The sequence of installation that I did was:

Option 2: Install LISVAP python module using pip tool

1. Creating a conda enviromment for LISVAP conda create --name lisvap python=3.8 --no-default-packages source activate lisvap

     By using just git clone and git lfs clone, but both commands operate exactly the same.

   WARNING: 'git lfs clone' is deprecated and will not be updated with new flags from 'git clone' 'git clone' has been updated in upstream Git to have comparable speeds to 'git lfs clone'

2. Installing lisvap pip install lisflood-lisvap lisvap

   LisvapPy - Lisvap (Global) using pcraster Python framework
   Version      : 1.0.4
   Last updated : 27/03/2020
   Status       : Operation
   Authors      : Peter Burek, Johan van der Knijff, Ad de Roo
   Maintainers  : Goncalo Gomes, Domenico Nappo, Valerio Lorini, Lorenzo Mentaschi, Lorenzo Alfieri
   ...

3. Installing and checking pcraster conda install -c conda-forge pcraster In a python environment run import pcraster as pcr pcr.__version__

4. Running lisvap examples from /my_system/lisflood-lisvap/tests/data/tests_efas.xml -v -t lisvap mysettings.xml -v -t

Option 3: Clone the repository from github

1. clone LISVAP from the respository git clone https://github.com/ec-jrc/lisflood-lisvap.git cd lisflood-lisvap

    I tried two ways, by using `git clone` and `git lfs clone`, but both commands operate exactly the same.

   WARNING: 'git lfs clone' is deprecated and will not be updated with new flags from 'git clone' 'git clone' has been updated in upstream Git to have comparable speeds to 'git lfs clone'

2. Creating a conda enviromment for lisvap conda create --name lisvap python=3.8 --no-default-packages source activate lisvap

3. Installing and checking lisvap pip install -r requirements.txt cd /my_system/lisflood-lisvap/src/ python lisvap1.py

   LisvapPy - Lisvap (Global) using pcraster Python framework
   Version      : 1.0.4
   Last updated : 27/03/2020
   Status       : Operation
   Authors      : Peter Burek, Johan van der Knijff, Ad de Roo
   Maintainers  : Goncalo Gomes, Domenico Nappo, Valerio Lorini, Lorenzo Mentaschi, Lorenzo Alfieri
   ...

4. Installing and checking pcraster conda install -c conda-forge pcraster in a python environment run import pcraster as pcr pcr.__version__

   4.3.3

5. Running LISVAP from scr folder python lisvap1.py /my_system/lisflood-lisvap/tests/data/tests_efas.xml -v -t

Links: [1] https://ec-jrc.github.io/lisflood-lisvap/3_LISVAP_installation/

[markdurel]

Dear @gnrgomes, Thank you for your prompt response. I actually followed the recommendations. Ok, I'm currently having the same problem @EdgarEspitia

========================== LISFLOOD ERROR ============================= /home/mak/projet_lisvap/lisflood-lisvap/tests/data/projet_lisvap/lisflood-lisvap/basemaps/area.nc (pcraster) (LISVAP) (base) mak@Leopard:~/projet_lisvap/lisflood-lisvap/src$

[gnrgomes]

Dear Mark, You most probably have the wrong path configured in your settings file. Could you please correct the path to your file area.nc in the settings? In the logs you have: /home/mak/projet_lisvap/lisflood-lisvap/tests/data/projet_lisvap/lisflood-lisvap/basemaps/area.nc Most probably what you should have is: /home/mak/projet_lisvap/lisflood-lisvap/basemaps/area.nc

Meanwhile, I will check what seems to be wrong with the download of the base maps from github.

[gnrgomes]

Dear @markdurel and @EdgarEspitia , I just cloned the repository and I was able to get all the base map files. Did you install git LFS before cloning the repository? Please, follow the instructions for your operating system on the page bellow. https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage

[markdurel]

@gnrgomes Thank you!

I would like to know what parameter file is it? Because the file in which I changed the path is "tests_efas.xml" and here is the syntax I used at this level: textvar name="PathBaseMapsIn" value="$(SettingsDir)/basemaps"> However, by executing the command "$ python lisvap1.py /../../tests_efs.xml " I don't understand why instead of having /home/mak/projet_lisvap/lisflood-lisvap/basemaps/area.nc I have the same error with yet another different path, here is the error : ========================== LISFLOOD ERROR ====================== ======= /home/mak/project_lisvap/lisflood-lisvap/tests/data/basemaps/area.nc (pcraster) (LISVAP) (base) mak@Leopard:~/lisvap_project/lisflood-lisvap/src$ I did several path replacements and found that the /tests/data/ part appears in all errors.

[markdurel]

Dear @gnrgomes Also regarding cloning the repository I had done it using github and was able to get all base map files. You can see it on the screenshot

[EdgarEspitia]

Dear @gnrgomes,

Thank you very much. I followed your instructions, so LISVAP is working properly in my system!

The problem was solved by installing and initializing git for large files before cloning LISVAP to my system, after that just follow the installation instructions for LISVAP.

To summarize, the installation was
Option 3: Clone the repository from github

1. Installing/initializing git for large file
   git lfs install
2. clone LISVAP from the respository git clone https://github.com/ec-jrc/lisflood-lisvap.git cd lisflood-lisvap
3. Creating a conda enviromment for lisvap conda create --name lisvap python=3.8 --no-default-packages source activate lisvap
4. Installing and checking lisvap pip install -r requirements.txt
5. Installing and checking pcraster conda install -c conda-forge pcraster
6. Running LISVAP from scr folder python lisvap1.py /my_system/lisflood-lisvap/tests/data/tests_efas.xml -v -t

[markdurel]

Good evening @EdgarEspitia I want to thank you for sharing the procedure. I admit that it was not easy, but I managed to do the test provided by the model. I would like to have your email so that we work in syndiosis Well, @gnrgomes I would like to know how to read output files (e0, es, et)

[gnrgomes]

Good morning @markdurel you can read the output files the same way you read the input files. Meaning with any GIS software or Panoply or even using python through the netCDF4 library.

[EdgarEspitia]

Dear @markdurel,

Check this script (read_e0.zip) in python, It is an example to explore the outputs of LISVAP. You need to install python and some packages.

Another way, the easier one, is by using Panoply. Check this website: https://www.giss.nasa.gov/tools/panoply/

[markdurel]

Good morning Thanks Dear @EdgarEspitia , the Panoply tool works very well and the $ncview filename.nc command also allows the visualization of netcdf files. Dear @gnrgomes, I share with you the difficulties I am facing. Well, this is the implementation of the model. I still can't find the input data for my study area. As I told you, the model runs well on my computer and I could see it with the tests offered on the official documentation.

[gnrgomes]

Dear @markdurel, Your input data and static maps will be provided by my colleague Stefania. She will contact you soon, via email with more details.

[diksha922]

Hello Sir/ Madam,

I have followed all the steps as mentioned in the LISVAP website for running the test cases (LINUX system) but facing the following error:

python lisvap1.py /home/diksha/lisflood-lisvap/tests/data/tests_efas.xml -v -t Lisvap 1.2.6 16/03/2023

(C) Disaster Risk Management Knowledge Centre Joint Research Centre of the European Commission Units E1,D2 I-21020 Ispra (Va), Italy

Start date: 02/01/1990 06:00 (1) - End date: 15/01/1990 06:00 (14) Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 2 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 3 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 4 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 5 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 6 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 7 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 8 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 9 Trying to access file /home/diksha/lisflood-lisvap/basemaps/area.nc: attempt n. 10 Traceback (most recent call last): File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 61, in loadsetclone iter_setclone_pcraster(filename) File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 377, in iter_setclone_pcraster return remote_input_access(pcraster.pcraster.setclone, file_path) File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 394, in remote_input_access obj = function(file_path) TypeError: Cannot open '/home/diksha/lisflood-lisvap/basemaps/area.nc'. Note: only the PCRaster file format is supported as input argument.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 394, in remote_input_access obj = function(file_path) File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 366, in access_function return Dataset(path, mode, **kwargs) File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.init File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success OSError: [Errno -51] NetCDF: Unknown file format: '/home/diksha/lisflood-lisvap/basemaps/area.nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "lisvap1.py", line 97, in sys.exit(main()) File "lisvap1.py", line 93, in main lisvapexe(lissettings) File "lisvap1.py", line 63, in lisvapexe lisvap_model = LisvapModel() File "/home/diksha/lisflood-lisvap/src/lisvap/lisvapinitial.py", line 49, in init self.MaskMap = loadsetclone('MaskMap') File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 67, in loadsetclone nf1 = iter_open_netcdf(filename, 'r') File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 367, in iter_open_netcdf return remote_input_access(access_function, file_path) File "/home/diksha/lisflood-lisvap/src/lisvap/utils/readers.py", line 399, in remote_input_access raise Exception("Cannot access file {0}!\nNetwork down for too long OR bad root directory {1}!".format(file_path, root)) Exception: Cannot access file /home/diksha/lisflood-lisvap/basemaps/area.nc! Network down for too long OR bad root directory /home/diksha/lisflood-lisvap!

It is requested to kindly help me out so I can run the test cases properly in my LINUX system.

[gnrgomes]

Dear @diksha922 , Let me try to understand what might be the issue. We'll go step by step.

1) Could you please control the size of the file?

ls -lah /home/diksha/lisflood-lisvap/basemaps/

2) Can you open/read the file /home/diksha/lisflood-lisvap/basemaps/area.nc ? For example using ncinfo or even Panoply or does it give any error while opening?

Bellow you have my results for the questions above for you to compare.

[diksha922]

Sir/ Madam,

It seems that the input and setting files cloned from the git is having some issues as it is showing .nc files as ASCII text file type.

Please resolve the issue.

[diksha922]

Sir/ Madam,

I have tried to run the test cases in LINUX machine after downloading the individual input and setting files from the github.

Now the test cases for - cordex, glofas and efas are running fine and generating the outputs. However, the other three cases - efas_1arcmin, efas_1arcmin_yearly and efas_1arcmin_360days_calendar are showing "killed" error (Screenshot attached).

It is requested to kindly help me out in running these test cases on my LINUX machine.

[gnrgomes]

Dear @diksha922 , There is no issue to solve from my side. Please install git LFS first.

You can follow the instructions here: 1) https://github.com/ec-jrc/lisflood-lisvap/issues/46#issuecomment-1106204950 2) https://github.com/ec-jrc/lisflood-lisvap/issues/46#issuecomment-1108005632

I hope now you can proceed.

For your other comment about helping you setting up the maps for your area of interest, I will see if Monday we can point you out in the right path to achieve your goal.

Have a nice weekend.

[diksha922]

Sir/ Madam,

I have followed all the above steps but these test cases are not running as mentioned in the previous attached comments.

Please help me in this.

[markdurel]

Dear @diksha922, you are asked to install git LFS. After that you will have to clone the LISVAP repository from git LFS.

Can you share with us the current error that appears during your recent executions?

[diksha922]

Sir, I have installed git LFS and after that I have cloned the LISVAP repository from git LFS.

I have tried to run the test cases available in the LISVAP folder. The test cases for - cordex, glofas and efas are running fine and generating the outputs. However, the other three cases - efas_1arcmin, efas_1arcmin_yearly and efas_1arcmin_360days_calendar are showing "killed" error (highlighted in the Screenshot attached) and are not completing simulation & generating any outputs.

Please help me.

[gnrgomes]

Dear @diksha922 , I installed a fresh environment and was able to run on my linux virtual machine (Ubuntu) all the tests without any issues. Since most of the operations are done in memory (RAM) I suspect you have very little RAM assigned to your VM. Could you please check how much RAM you have on your VM? If possible increase the RAM on your VM configuration. I checked that running all the tests using the command pytest it uses at its peak around 2.4 GB of RAM, and you also need memory to run your linux system. I would say, you need at least 5GB of RAM configured on your VM, but you can experiment with other values to find one that suits your needs. Best regards.

[diksha922]

Thank you sir for the information.

Sir I have increased the size of RAM on my VM configuration from 2.3 GB to 3.5 GB. Now the test cases are running fine.

Thank you once again.