iris=3.0.0 needs cftime<=1.2.1 in a conda environment

[valeriupredoi]

📰 Custom Issue

Hey guys, congratulations for releasing iris=3.0.0 :partying_face: :tada: We have tried to install it in the ESMValCore environment so we can grab it for our upcoming release but hit a stumbling block - it needs cftime lower or equal to (an old version) 1.2.1, see comment - you see, we had previously pinned cftime to >=1.3.0 because we noticed some test fails due to new years conventions when years < 1000 ie they write "0999" instead of " 999", anyways, we can fix those tests, but the more unnerving bit is the environment will forever be stuck with an old cftime - are there any plans to unpin that from your environment requirements soon? Cheers :beer:

PS: here is our fully solved environment as lady Conda on Circle CI solved it:

name: esmvaltool
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - _r-mutex=1.0.1=anacondar_1
  - antlr-python-runtime=4.7.2=py39hde42818_1002
  - atk-1.0=2.36.0=h3371d22_4
  - binutils=2.35.1=hdd6e379_1
  - binutils_impl_linux-64=2.35.1=h193b22a_1
  - binutils_linux-64=2.35=hc3fd857_29
  - bokeh=2.2.3=py39hf3d152e_0
  - boost-cpp=1.74.0=h9359b55_0
  - bwidget=1.9.14=ha770c72_0
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.17.1=h36c2ea0_0
  - c-compiler=1.1.1=h516909a_0
  - ca-certificates=2020.12.5=ha878542_0
  - cairo=1.16.0=h488836b_1006
  - cartopy=0.18.0=py39h32a4312_7
  - certifi=2020.12.5=py39hf3d152e_1
  - cf-units=2.1.4=py39h1dff97c_2
  - cfitsio=3.470=h2e3daa1_7
  - cftime=1.2.1=py39h1dff97c_1
  - click=7.1.2=pyh9f0ad1d_0
  - cloudpickle=1.6.0=py_0
  - compilers=1.1.1=0
  - curl=7.71.1=he644dc0_8
  - cxx-compiler=1.1.1=hc9558a2_0
  - cycler=0.10.0=py_2
  - cytoolz=0.11.0=py39h3811e60_3
  - dask=2021.1.1=pyhd8ed1ab_0
  - dask-core=2021.1.1=pyhd8ed1ab_0
  - distributed=2021.1.1=py39hf3d152e_0
  - esmf=8.0.1=mpi_mpich_h61b2105_102
  - esmpy=8.0.1=mpi_mpich_py39h432445a_102
  - expat=2.2.9=h9c3ff4c_3
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=2.001=hab24e00_0
  - font-ttf-source-code-pro=2.030=hab24e00_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.13.1=hba837de_1004
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fortran-compiler=1.1.1=he991be0_0
  - freetype=2.10.4=h0708190_1
  - freexl=1.0.5=h516909a_1002
  - fribidi=1.0.10=h36c2ea0_0
  - fsspec=0.8.5=pyhd8ed1ab_0
  - gcc_impl_linux-64=7.5.0=hda68d29_13
  - gcc_linux-64=7.5.0=he2a3fca_29
  - gdk-pixbuf=2.42.2=h0c95a7a_2
  - geos=3.8.1=he1b5a44_0
  - geotiff=1.6.0=h5d11630_3
  - gettext=0.19.8.1=h0b5b191_1005
  - gfortran_impl_linux-64=7.5.0=h56cb351_18
  - gfortran_linux-64=7.5.0=ha081f1e_29
  - giflib=5.2.1=h36c2ea0_2
  - glib=2.66.4=hc4f0c31_2
  - glib-tools=2.66.4=hc4f0c31_2
  - graphite2=1.3.13=h58526e2_1001
  - graphviz=2.42.3=h6939c30_2
  - gsl=2.6=he838d99_2
  - gtk2=2.24.33=hab0c2f8_0
  - gts=0.7.6=h64030ff_2
  - gxx_impl_linux-64=7.5.0=h64c220c_13
  - gxx_linux-64=7.5.0=h547f3ba_29
  - harfbuzz=2.7.2=ha5b49bf_1
  - hdf4=4.2.13=h10796ff_1004
  - hdf5=1.10.6=mpi_mpich_hc41e3f9_1011
  - hdfeos2=2.20=h64bfcee_1000
  - hdfeos5=5.1.16=hb4ce793_7
  - heapdict=1.0.1=py_0
  - icu=67.1=he1b5a44_0
  - iris=3.0.0=py39hf3d152e_0
  - jasper=1.900.1=h07fcdf6_1006
  - jinja2=2.11.2=pyh9f0ad1d_0
  - jpeg=9d=h36c2ea0_0
  - json-c=0.13.1=hbfbb72e_1002
  - kealib=1.4.14=h0042707_0
  - kernel-headers_linux-64=2.6.32=h77966d4_13
  - kiwisolver=1.3.1=py39h1a9c180_1
  - krb5=1.17.2=h926e7f8_0
  - lcms2=2.11=hcbb858e_1
  - ld_impl_linux-64=2.35.1=hea4e1c9_1
  - libblas=3.9.0=7_openblas
  - libcblas=3.9.0=7_openblas
  - libcurl=7.71.1=hcdd3856_8
  - libdap4=3.20.6=h1d1bd15_1
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libffi=3.3=h58526e2_2
  - libgcc-ng=9.3.0=h2828fa1_18
  - libgdal=3.1.4=h50e41a3_1
  - libgfortran-ng=7.5.0=h14aa051_18
  - libgfortran4=7.5.0=h14aa051_18
  - libglib=2.66.4=h748fe8e_2
  - libgomp=9.3.0=h2828fa1_18
  - libiconv=1.16=h516909a_0
  - libkml=1.3.0=h74f7ee3_1012
  - liblapack=3.9.0=7_openblas
  - libnetcdf=4.7.4=mpi_mpich_hdef422e_7
  - libnghttp2=1.41.0=h8cfc5f6_2
  - libopenblas=0.3.12=pthreads_hb3c22a3_1
  - libpng=1.6.37=h21135ba_2
  - libpq=12.3=h255efa7_3
  - libspatialite=5.0.0=h4dde289_0
  - libssh2=1.9.0=hab1572f_5
  - libstdcxx-ng=9.3.0=h6de172a_18
  - libtiff=4.2.0=hdc55705_0
  - libtool=2.4.6=h58526e2_1007
  - libuuid=2.32.1=h7f98852_1000
  - libwebp=1.1.0=h76fa15c_4
  - libwebp-base=1.1.0=h36c2ea0_3
  - libxcb=1.13=h7f98852_1003
  - libxml2=2.9.10=h68273f3_2
  - locket=0.2.0=py_2
  - lz4-c=1.9.3=h9c3ff4c_0
  - make=4.3=hd18ef5c_1
  - markupsafe=1.1.1=py39h3811e60_3
  - matplotlib-base=3.3.3=py39h2fa2bec_0
  - mpi=1.0=mpich
  - mpi4py=3.0.3=py39h6438238_4
  - mpich=3.3.2=h846660c_5
  - msgpack-python=1.0.2=py39h1a9c180_1
  - ncl=6.6.2=h087f628_23
  - ncurses=6.2=h58526e2_4
  - netcdf-fortran=4.5.3=mpi_mpich_h3923e1a_1
  - netcdf4=1.5.5.1=nompi_py39h36800e2_101
  - numpy=1.19.5=py39hdbf815f_1
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hf7af979_0
  - openssl=1.1.1i=h7f98852_0
  - packaging=20.8=pyhd3deb0d_0
  - pandas=1.2.1=py39hde0f152_0
  - pango=1.42.4=h69149e4_5
  - partd=1.1.0=py_0
  - pcre=8.44=he1b5a44_0
  - pcre2=10.35=h032f7d1_2
  - pillow=8.1.0=py39h2bb83ca_1
  - pip=21.0=pyhd8ed1ab_0
  - pixman=0.38.0=h516909a_1003
  - poppler=0.89.0=h2de54a5_5
  - poppler-data=0.4.10=0
  - postgresql=12.3=hc2f5b80_3
  - proj=7.1.1=h966b41f_3
  - psutil=5.8.0=py39h3811e60_1
  - pthread-stubs=0.4=h36c2ea0_1001
  - pyke=1.1.1=pyhd8ed1ab_1004
  - pyparsing=2.4.7=pyh9f0ad1d_0
  - pyshp=2.1.3=pyh44b312d_0
  - python=3.9.1=hffdb5ce_5_cpython
  - python-dateutil=2.8.1=py_0
  - python-stratify=0.1.1=py39hce5d2b2_1003
  - python-xxhash=2.0.0=py39h3811e60_1
  - python_abi=3.9=1_cp39
  - pytz=2020.5=pyhd8ed1ab_0
  - pyyaml=5.4.1=py39h3811e60_0
  - r-base=4.0.2=he766273_1
  - r-yaml=2.2.1=r40hcfec24a_1
  - readline=8.0=he28a2e2_2
  - scipy=1.5.3=py39hf3f25e7_0
  - sed=4.8=he412f7d_0
  - setuptools=49.6.0=py39hf3d152e_3
  - shapely=1.7.1=py39hcbe974e_1
  - six=1.15.0=pyh9f0ad1d_0
  - sortedcontainers=2.3.0=pyhd8ed1ab_0
  - sqlite=3.34.0=h74cdb3f_0
  - sysroot_linux-64=2.12=h77966d4_13
  - tblib=1.6.0=py_0
  - tiledb=2.1.6=h91fcb0e_1
  - tk=8.6.10=h21135ba_1
  - tktable=2.10=hb7b940f_3
  - toolz=0.11.1=py_0
  - tornado=6.1=py39h3811e60_1
  - typing_extensions=3.7.4.3=py_0
  - tzcode=2021a=h7f98852_0
  - tzdata=2021a=he74cb21_0
  - udunits2=2.2.27.27=h360fe7b_0
  - wheel=0.36.2=pyhd3deb0d_0
  - xerces-c=3.2.3=hfe33f54_1
  - xorg-imake=1.0.7=0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h516909a_0
  - xorg-libsm=1.2.3=h84519dc_1000
  - xorg-libx11=1.6.12=h516909a_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxaw=1.0.13=h516909a_1002
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h516909a_0
  - xorg-libxmu=1.1.3=h516909a_0
  - xorg-libxpm=3.5.13=h516909a_0
  - xorg-libxrender=0.9.10=h516909a_1002
  - xorg-libxt=1.1.5=h516909a_1003
  - xorg-makedepend=1.0.6=he1b5a44_1
  - xorg-renderproto=0.11.1=h14c3975_1002
  - xorg-xextproto=7.3.0=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xxhash=0.8.0=h7f98852_3
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h516909a_0
  - zict=2.0.0=py_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.4.8=ha95c52a_1
  - pip:
    - apipkg==1.5
    - attrs==20.3.0
    - chardet==4.0.0
    - click-plugins==1.1.1
    - cligj==0.7.1
    - coverage==5.4
    - cython==0.29.21
    - decorator==4.4.2
    - esmvalcore==2.1.0
    - esmvaltool-sample-data==0.0.3
    - execnet==1.7.1
    - fiona==1.8.18
    - fire==0.4.0
    - flake8==3.8.4
    - idna==2.10
    - iniconfig==1.1.1
    - isodate==0.6.0
    - latexcodec==2.0.1
    - lxml==4.6.2
    - mccabe==0.6.1
    - munch==2.5.0
    - nc-time-axis==1.2.0
    - networkx==2.5
    - pluggy==0.13.1
    - prov==2.0.0
    - py==1.10.0
    - pybtex==0.24.0
    - pycodestyle==2.6.0
    - pydot==1.4.1
    - pyflakes==2.2.0
    - pytest==6.2.2
    - pytest-cov==2.11.1
    - pytest-env==0.6.2
    - pytest-flake8==1.0.7
    - pytest-forked==1.3.0
    - pytest-html==3.1.1
    - pytest-metadata==1.11.0
    - pytest-mock==3.5.1
    - pytest-xdist==2.2.0
    - rdflib==5.0.0
    - requests==2.25.1
    - termcolor==1.1.0
    - toml==0.10.2
    - urllib3==1.26.2
    - yamale==2.2.0
prefix: /opt/conda/envs/esmvaltool

[trexfeathers]

This is complicated by the fact that two of Iris' dependencies - cf-units and nc-time-axis - are also 'broken' by cftime 1.3, but no resource has so far been committed to sorting this nor making new releases once fixed. I would very much like to get this sorted in the near future!

[valeriupredoi]

cheers @trexfeathers - we are happy that so far only this pin was needed and are proceeding to fix the tests that the pin breaks, further on things look good :+1:

[bjlittle]

Hey @valeriupredoi

Yeah, we've pinned cftime<1.3.0 for iris master and v3.0.0. As @trexfeathers mentions we're keen to fix this, but may require a bit of effort on several packages - hence the pin.

However, if we do unpin, it may result in an iris v3.0.1 release - on the other hand, we may just fold that goodness into iris v3.1.0, which is scheduled for the summer 2021 (we'll firm the proposed release date soon)

But if you can work around this pin, that would be appreciated :wink:

[valeriupredoi]

cheers @bjlittle - we completely understand the issues with dependency packages that evolve differentially, in other words, they need to get their conflicts act together so we or you have a healthy up-to-date environment too :rofl: cftime=1.2.1 as it is pinned now in our environment to accommodate iris=3.0 has not given us any headaches and it's been around for a while now, so I think this is a non-issue for us. I'll have a look at the actual iris environment myself, but I doubt I can fix anything that you guys didn't :grin: But will let you know if I manage anything!

[rcomer]

I have opened https://github.com/SciTools/cf-units/pull/168 for getting cf_units up to speed with cftime v1.5 🙂

Edit: and it looks like we have a new contributor who has done the same for nc-time-axis: https://github.com/SciTools/nc-time-axis/pull/59 🎉

[trexfeathers]

We're getting there. As indicated we are dependent on nc-time-axis and cf-units first, so those are getting the initial attention.

[valeriupredoi]

hey guys, as I see things now cf-units is doing fine at 3.0.1 (cheers, Ruth!) but nc-time-axis is still at the ye olde 1.2.0 - what does the status: blocked mean for this issue? Lemme know if I can help :+1:

[rcomer]

We are almost there: once #4222 is merged and we release Iris v3.0.4, we will have a full set of Scitools packages that work with cftime v1.5. In the meantime, we have

• Latest Iris version requires cftime < 1.3
• Latest nc-time-axis requires cftime >= 1.5

So these are incompatible 😢

Latest cf_units works with the older or newer cftime 👍

[valeriupredoi]

cheers muchly @rcomer - apols for the late reply, too hot to think anything more complex than "water (or beer)" :sweat:

[bjlittle]

@valeriupredoi I'm now closing this issue as iris 3.0.4 unpins cftime, finally allowing you to use the latest iris along with the latest versions of nc-time-axis, cf-units and cftime :+1:

iris 3.0.4 is already on PyPI, and I'm currently in the process of building the conda recipe for conda-forge... so that'll land imminently.

Thanks @trexfeathers, @rcomer and @pp-mo for all helping to get this across the line :partying_face: :beers: :rocket: