rf_glofas breaks climada_petals installation on Euler

[emanuel-schmid]

The current installation environment on Euler is not compatible with the python packages pymrio and rioxarray. Installing those with pip leads to re-installation of numpy, pandas, numba, and other packages. Eventually this all leads to an incompatibility with shapely 2.0.

workaround

Until the issue is solved the suggested workaround is to remove these rf_glofas dependencies from the setup.py file before running pip install -e <path to climada_petals repository>:

    ...
    keywords='climate adaptation',

    install_requires=[
        'climada>=4.1',
        'cdsapi',
        'ruamel.yaml',
        'scikit-image',
        'xesmf',
        #"pymrio",
        #'rioxarray',
        'osm-flex',
    ],

    packages=find_namespace_packages(include=['climada_petals*']),
    ...

[peanutfun]

The "latest" update of rioxarray changing its dependencies was in January, and the changed dependencies were

DEP: Support Python 3.10-3.12, rasterio 1.3+, pyproj 3.3+, xarray 2022.3.0+ & numpy 1.23+

The issue here seems to be that it requires numpy >=1.23, but the Python 3.10 environment on Euler only provides numpy 1.22. Pip will then locally install a new version, which does not work in the Euler setup. The workaround should be to set rioxarray<=0.15.0 in the dependencies for now.

pymrio is a module required for the SupplyChain module and thus unrelated to the rf_glofas module, see #66. @aleeciu @spjuhel, do you have any insights into dependency updates of pymrio?

[spjuhel]

I know they are still actively developing, and Konstantin Stadler, the main developer, once got in touch with me, so I can contact him.

But what's the actual problem with pymrio? To my understanding, it does not require numpy > 1.22 ?

https://github.com/IndEcol/pymrio/blob/944971764a627cb3d0142256720007c76cbfc36a/setup.py#L23C4-L34C7

[peanutfun]

@spjuhel I think the issue with pymrio is pyarrow: Euler only provides pyarrow v9.0.0 for Python 3.10, but pymrio requires >= 11.0.

[peanutfun]

@emanuel-schmid Would it be an option to upgrade to a Python 3.11 or 3.12 environment on Euler? Would the pre-installed packages meet these requirements there?

[spjuhel]

Note that pyarrow 9.0.0 is pretty outdated. For instance dask (optimized pandas DataFrame package) will soon drop support for version below 14.0.0...

[peanutfun]

Of course, but the environment on Euler will not change, at least not the one of Python 3.10, as far as I understand. That's why we have to switch to a newer Python version on Euler, I think.

[emanuel-schmid]

👍 Indeed. I've installed climada 4.1 on Euler:

module load gcc/8.2.0 python/3.11.2 hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5
pip install climada==4.1
pip install climada_petals==4.1

worked for me. SIS/HPC will make climada 4.1 the pre-installed default package in the python 3.11 module on Euler. (climada 3.2 does not work with numpy causing trouble)

I'll update the Euler user guide and make a PR shortly.

[peanutfun]

@emanuel-schmid excellent work!