search

NCAS-CMS / cf-python

A CF-compliant Earth Science data analysis library
http://ncas-cms.github.io/cf-python
MIT License
116 stars 19 forks source link

CI: on one workflow, fix installation to make ESMF available #629

Open sadielbartholomew opened 1 year ago

sadielbartholomew commented 1 year ago

The esmf library isn't being installed properly for our test-source-dist CI workflow, so that it isn't available and therefore regridding capability depending on ESMPy is erroring out, for example as shown from this snippet in the logs from the recent job https://github.com/NCAS-CMS/cf-python/actions/runs/4551782556/jobs/8026279312?pr=627:

2023-03-29T08:26:16.2503387Z Run date: 2023-03-29 08:26:14.821170
2023-03-29T08:26:16.2503879Z Platform: Linux-5.15.0-1034-azure-x86_64-with-glibc2.10 
2023-03-29T08:26:16.2504206Z HDF5 library: 1.12.2 
2023-03-29T08:26:16.2504471Z netcdf library: 4.8.1 
2023-03-29T08:26:16.2505011Z udunits2 library: /usr/share/miniconda3/envs/cf-latest/lib/libudunits2.so.0 
2023-03-29T08:26:16.2505343Z ESMF: not available 
2023-03-29T08:26:16.2505728Z Python: 3.8.16 /usr/share/miniconda3/envs/cf-latest/bin/python
2023-03-29T08:26:16.2506241Z dask: 2023.3.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/dask/__init__.py
2023-03-29T08:26:16.2506817Z netCDF4: 1.6.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/netCDF4/__init__.py
2023-03-29T08:26:16.2507398Z psutil: 5.9.4 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/psutil/__init__.py
2023-03-29T08:26:16.2507988Z packaging: 23.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/packaging/__init__.py
2023-03-29T08:26:16.2509002Z numpy: 1.24.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/numpy/__init__.py
2023-03-29T08:26:16.2509565Z scipy: 1.10.1 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/scipy/__init__.py
2023-03-29T08:26:16.2510160Z matplotlib: 3.2.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/matplotlib/__init__.py
2023-03-29T08:26:16.2510727Z cftime: 1.6.2 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cftime/__init__.py
2023-03-29T08:26:16.2511303Z cfunits: 3.3.5 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfunits/__init__.py
2023-03-29T08:26:16.2512063Z cfplot: 3.2.0 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfplot/__init__.py
2023-03-29T08:26:16.2512681Z cfdm: 1.10.0.3 /usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfdm/__init__.py
2023-03-29T08:26:16.2513175Z cf: 3.14.1 /home/runner/tmp/test_cf-python/cf/__init__.py
2023-03-29T08:26:16.2513593Z 
2023-03-29T08:26:16.2513950Z 
2023-03-29T08:26:16.2514362Z ----------------------------------------------------------------------
2023-03-29T08:26:16.2515213Z Ran 12 tests in 0.958s
2023-03-29T08:26:16.2515424Z 
2023-03-29T08:26:16.2515510Z OK
2023-03-29T08:26:17.7718166Z Traceback (most recent call last):
2023-03-29T08:26:17.7728610Z   File "test_RegridOperator.py", line 10, in <module>
2023-03-29T08:26:17.7729478Z     class RegridOperatorTest(unittest.TestCase):
2023-03-29T08:26:17.7730287Z   File "test_RegridOperator.py", line 13, in RegridOperatorTest
2023-03-29T08:26:17.7731409Z     r = src.regrids(dst, "linear", return_operator=True)
2023-03-29T08:26:17.7732380Z   File "/home/runner/tmp/test_cf-python/cf/decorators.py", line 71, in precede_with_kwarg_deprecation_check
2023-03-29T08:26:17.7732831Z     operation_method_result = operation_method(self, *args, **kwargs)
2023-03-29T08:26:17.7733518Z   File "/usr/share/miniconda3/envs/cf-latest/lib/python3.8/site-packages/cfdm/decorators.py", line 44, in inplace_wrapper
2023-03-29T08:26:17.7734001Z     processed_copy = operation_method(self, *args, **kwargs)
2023-03-29T08:26:17.7734514Z   File "/home/runner/tmp/test_cf-python/cf/field.py", line 14169, in regrids
2023-03-29T08:26:17.7734812Z     return regrid(
2023-03-29T08:26:17.7735269Z   File "/home/runner/tmp/test_cf-python/cf/regrid/regrid.py", line 362, in regrid
2023-03-29T08:26:17.7735642Z     ESMF_manager = ESMF_initialise()  # noqa: F841
2023-03-29T08:26:17.7736172Z   File "/home/runner/tmp/test_cf-python/cf/regrid/regrid.py", line 1169, in ESMF_initialise
2023-03-29T08:26:17.7736516Z     raise RuntimeError(
2023-03-29T08:26:17.7736861Z RuntimeError: Regridding will not work unless the ESMF library is installed
2023-03-29T08:26:17.9028758Z + test_result_status=1
2023-03-29T08:26:17.9034545Z + set -x
2023-03-29T08:26:17.9035492Z + exit 1
2023-03-29T08:26:17.9050536Z ##[error]Process completed with exit code 1.

ESMF did get installed OK for this workflow before, so something has changed. I don't have time to look into this before Easter, but note for that job, as an example, from a quick search through the logs, the library does get downgraded, which probably relates to the issue:

2023-03-29T08:21:43.8814896Z The following packages will be SUPERSEDED by a higher-priority channel:
2023-03-29T08:21:43.8815149Z 
2023-03-29T08:21:43.8815609Z   esmpy              conda-forge/noarch::esmpy-8.4.1-pyhc1~ --> conda-forge/linux-64::esmpy-8.4.0-nompi_py38h2b78397_2 
2023-03-29T08:21:43.8815916Z 
2023-03-29T08:21:43.8816072Z The following packages will be DOWNGRADED:
2023-03-29T08:21:43.8816273Z 
2023-03-29T08:21:43.8816563Z   esmf                               8.4.1-nompi_h5b4e5e4_2 --> 8.4.0-nompi_hdb2cfa9_4 
2023-03-29T08:21:43.8817029Z   hdf4                                    4.2.15-h501b40f_6 --> 4.2.15-h9772cbc_5 
2023-03-29T08:21:43.8817524Z   hdf5                            1.14.0-nompi_hb72d44e_103 --> 1.12.2-nompi_h4df4325_101 
2023-03-29T08:21:43.8818006Z   icu                                       72.1-hcb278e6_0 --> 67.1-he1b5a44_0 
2023-03-29T08:21:43.8818447Z   lcms2                                     2.15-haa2dc70_1 --> 2.15-hfd0df8a_0 
2023-03-29T08:21:43.8818920Z   libdeflate                                1.18-h0b41bf4_0 --> 1.17-h0b41bf4_0 
2023-03-29T08:21:43.8819414Z   libnetcdf                        4.9.2-nompi_hf3f8848_102 --> 4.8.1-nompi_h21705cb_104 
2023-03-29T08:21:43.8819901Z   libtiff                                  4.5.0-ha587672_6 --> 4.5.0-h6adf6a1_2 
2023-03-29T08:21:43.8820381Z   matplotlib-base                      3.7.1-py38hd6c3c57_0 --> 3.2.2-py38h5d868c9_1 
2023-03-29T08:21:43.8820920Z   netcdf-fortran                   4.6.0-nompi_h3142581_105 --> 4.6.0-nompi_he1eeb6f_102 
2023-03-29T08:21:43.8821465Z   netcdf4                      1.6.3-nompi_py38hd31e641_102 --> 1.6.0-nompi_py38h6b4b75c_103 
2023-03-29T08:21:43.8822115Z   pillow                               9.4.0-py38h961100d_2 --> 9.4.0-py38hde6dc18_1
sadielbartholomew commented 1 year ago

Probably this relates to the following backward incompatible changes at ESMF release 8.4.0, namely as covered in the relevant Changelog entry:

The Python module name for ESMPy was changed in this release from “ESMF” to “esmpy” for better alignment with Python Enhancement Proposal (PEP) guidelines. This change will require user code changes! ESMPy users will have to do a global search and replace on their scripts to adjust to the module name change. Also notice that the internal ESMF source tree directory structure has changed, moving ./src/addon/ESMPy to ./src/addon/esmpy. This might affect anybody keeping an ESMF repository clone, and maintaining scripts that depend on the internal directory naming.

davidhassell commented 4 months ago

Hi @sadielbartholomew - is this closable, now? Thanks

sadielbartholomew commented 4 months ago

Yes, please close this, though there are other aspects of the CI workflows I need to update at this point and am yet to find the time to do so. Shortly, I shall try to get it done.