gdal for PgdInput

[iljamal]

I've ran module load gdal/3.6.2 pip install pygdal=="gdal-config --version.* before and after entering poetry shell and runing deode demo case but still facing the gdal issues during ecflow execution

ImportError: Cannot use the installed gdal library, or there is no gdal library installed. If you have not installed it, you may want to try running 'pip install pygdal=="gdal-config --version.*"' or, if you use conda, 'conda install -c conda-forge gdal'.

Is there somewhere some module left unloaded?

[FlorianW-ZAMG]

On ATOS you should only need to load the gdal library with module load gdal . This would load gdal Version 3.8.4, there is not need to do pip install gdal.... on ATOS. Can you just try to load the gdal module and make sure you remove the local installation if you have one?

[iljamal]

can confirm that there is nothing in pip list | grep -i pygdal pip list | grep -i gdal the modules loaded module list Currently Loaded Modules: 1) gcc/8.5.0 3) cdo/2.4.0 5) ecmwf-toolbox/2024.04.0.0 7) hdf5/1.14.3 9) ecflow/5.12.4 11) python3/3.10.10-01 2) prgenv/gnu 4) nco/5.1.9 6) openmpi/4.1.5.4 8) netcdf4/4.9.2 10) gdal/3.8.4

And ecflow logs abort due to missing gdal

Is there a way to check the gdal dependencies /availability?

[uandrae]

Do you still have this this problem?

[iljamal]

Yes for me steps ..

module purge module load python3/3.10.10-01 module load ecflow export PATH="$HOME/.local/bin:$PATH" export ECF_HOST=ecflow-gen-$USER-001 module load gdal cd Deode-Workflow/ poetry shell deode case ?deode/data/config_files/configurations/cy48t3_arome -o cy48t3_arome.toml --start-suite

will not get past the gdalinput sequence

[draelsaid]

I've removed this line from the documentation because, as Florian says, ml gdal should be enough (on ATOS). We haven't done something equivalent for LUMI yet, but this line doesn't help on LUMI either.

[iljamal]

right, probbably some kind of python environment issues then

[uandrae]

I did the following for my user, trying to start from scratch and remove anything already installed

# Clean all previous installations
rm -rf ~/.local
module load python3/3.10.10-01
# Clean eventual previous poetry install
curl -sSL https://install.python-poetry.org | python3 - --uninstall
rm -rf ${HOME}/.cache/pypoetry/ ${HOME}/.local/bin/poetry ${HOME}/.local/share/pypoetry
# Download and install poetry
curl -sSL https://install.python-poetry.org | python3 -
cd Deode-Workflow
poetry install
poetry shell
module load gdal/3.6.2
# Testing the gdal import
python3
>>> from osgeo import gdal

You can also try using conda in a similar way that we did it here, not sure if this is more protected.

module load conda/23.11.0-0
cd Deode-Workflow
coda create -p .conda python=3.10.10 gdal=3.6.2 poetry
conda activate /etc/ecmwf/nfs/dh1_home_b/snh/deode/Deode-Workflow/.conda
deode -v
2024-06-19 04:44:41 | INFO     | Start deode v0.5.0 --> "deode -v"
deode v0.5.0
python3
>>> from osgeo import gdal
conda deactivate

Would be very interesting to know if this works for you as we need ways to create installations that doesn't interfere with already existing environments on a user level.

[iljamal]

Thanks for the guidelines.

I followed the first step, but this led to previous dependency issues with ecflow and I couldn't start the model at all.

It seems that the mamba-conda environment messed up the Python environment and the correct executable for deode. I thought this environment was necessary to communicate with Copernicus Marine Services, which is relevant for storm surge analysis. One needs to pay close attention to what it is linked to:

/etc/ecmwf/nfs/dh1_home_b/eeim_/Deode-Workflow/.venv/bin/python

In my case, some examples of non-working paths were:

• /usr/local/apps/python3/3.11.8-01/bin/python3
• /etc/ecmwf/nfs/dh1_home_b/eeim/mambaforge/bin/python3.10

Finally, after running: rm ~/mambaforge////poetry rm ~/mambaforge////deode mamba deactivate conda deactivate Just in case: rm -rf .venv/ Then reinstalling: poetry install for deode

The Python environment was finally correct:
.venv/bin/python -> /usr/local/apps/python3/3.10.10-01/bin/python3.10 Hope this helps..

Now the next, hopefully new, issue during runtime appeared :)

[draelsaid]

Great 😄