Readthedocs build is failing

[bouweandela]

Currently the readthedocs build is failing. see here for an example.

Failing command:

 mamba env create --quiet --name 2198 --file environment.yml

Output of all commands with error message at the end:

Read the Docs build information
Build id: 22020104
Project: esmvalcore
Version: 2198
Commit: 73751c2e0dfeabbaf2695f878ace70d488b238ef
Date: 2023-09-25T06:25:53.581054Z
State: finished
Success: False

[rtd-command-info] start-time: 2023-09-25T06:26:00.893948Z, end-time: 2023-09-25T06:26:01.986757Z, duration: 1, exit-code: 0
git clone --depth 1 https://github.com/ESMValGroup/ESMValCore .
Cloning into '.'...

[rtd-command-info] start-time: 2023-09-25T06:26:02.604987Z, end-time: 2023-09-25T06:26:03.404815Z, duration: 0, exit-code: 0
git fetch origin --force --prune --prune-tags --depth 50 pull/2198/head:external-2198
From https://github.com/ESMValGroup/ESMValCore
 * [new ref]         refs/pull/2198/head -> external-2198
 * [new tag]         v2.9.0              -> v2.9.0
 * [new tag]         v2.9.0rc1           -> v2.9.0rc1

[rtd-command-info] start-time: 2023-09-25T06:26:06.955497Z, end-time: 2023-09-25T06:26:07.198972Z, duration: 0, exit-code: 0
git checkout --force 73751c2e0dfeabbaf2695f878ace70d488b238ef
Note: switching to '73751c2e0dfeabbaf2695f878ace70d488b238ef'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 73751c2 Simplify code

[rtd-command-info] start-time: 2023-09-25T06:26:07.829633Z, end-time: 2023-09-25T06:26:07.893326Z, duration: 0, exit-code: 0
git clean -d -f -f

[rtd-command-info] start-time: 2023-09-25T06:26:08.465108Z, end-time: 2023-09-25T06:26:08.525992Z, duration: 0, exit-code: 0
cat .readthedocs.yaml
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the version of Python and other tools you might need
build:
  os: ubuntu-22.04
  tools:
    # updated and deployed from Aug 1, 2023
    python: "mambaforge-22.9"
  jobs:
    pre_create_environment:
      # update mamba just in case
      - mamba update --yes --quiet --name=base mamba
      - mamba --version
    post_create_environment:
      # use conda run executable wrapper to have all env variables
      - conda run -n ${CONDA_DEFAULT_ENV} mamba --version
      - conda run -n ${CONDA_DEFAULT_ENV} pip install . --no-deps

# Declare the requirements required to build your docs
conda:
  environment:
    environment.yml

# Build documentation in the doc directory with Sphinx
sphinx:
  configuration: doc/conf.py
  fail_on_warning: true

# If using Sphinx, optionally build your docs in additional formats such as PDF
formats:
  - pdf

[rtd-command-info] start-time: 2023-09-25T06:26:22.308719Z, end-time: 2023-09-25T06:26:22.399564Z, duration: 0, exit-code: 0
asdf global python mambaforge-22.9.0-3

[rtd-command-info] start-time: 2023-09-25T06:26:25.085717Z, end-time: 2023-09-25T06:26:54.627712Z, duration: 29, exit-code: 0
mamba update --yes --quiet --name=base mamba
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

[rtd-command-info] start-time: 2023-09-25T06:26:55.124139Z, end-time: 2023-09-25T06:26:55.689229Z, duration: 0, exit-code: 0
mamba --version
mamba 1.5.1
conda 22.9.0

[rtd-command-info] start-time: 2023-09-25T06:26:56.281196Z, end-time: 2023-09-25T06:26:56.342019Z, duration: 0, exit-code: 0
cat environment.yml
channels:
- conda-forge
- nodefaults
dependencies:
- cartopy
- cf-units
- cftime
- compilers
- dask
- dask-jobqueue
- distributed
- esgf-pyclient >=0.3.1
- esmpy !=8.1.0
- filelock
- fiona
- fire
- geopy
- humanfriendly
- importlib_metadata
- iris >=3.6.0
- iris-esmf-regrid >=0.7.0
- isodate
- jinja2
- libnetcdf !=4.9.1
- nc-time-axis
- nested-lookup
- netcdf4
- numpy !=1.24.3
- packaging
- pandas
- pillow
- pip !=21.3
- prov
- psutil
- py-cordex
- pybtex
- python >=3.9
- python-stratify >=0.3
- pyyaml
- requests
- scipy >=1.6
- shapely >=2.0.0
- yamale
- autodocsumm >=0.2.2
- ipython
- nbsphinx
- sphinx >=6.1.3
- pydata-sphinx-theme
- flake8
- mypy >=0.990
- pytest >=3.9,!=6.0.0rc1,!=6.0.0
- pytest-cov >=2.10.1
- pytest-env
- pytest-html !=2.1.0
- pytest-metadata >=1.5.1
- pytest-mypy
- pytest-mock
- pytest-xdist
- types-requests
- types-PyYAML
- codespell
- docformatter
- isort
- pre-commit
- prospector >=1.9.0
- yamllint
- yapf
- pip:
  - recommonmark
  - readthedocs-sphinx-ext
- mock
- pillow
- sphinx
- sphinx_rtd_theme
name: esmvaltool

[rtd-command-info] start-time: 2023-09-25T06:26:56.909864Z, end-time: 2023-09-25T06:27:16.759190Z, duration: 19, exit-code: 1
mamba env create --quiet --name 2198 --file environment.yml
/home/docs/.asdf/installs/python/mambaforge-22.9.0-3/lib/python3.10/site-packages/conda_package_streaming/package_streaming.py:19: UserWarning: zstandard could not be imported. Running without .conda support.
  warnings.warn("zstandard could not be imported. Running without .conda support.")
/home/docs/.asdf/installs/python/mambaforge-22.9.0-3/lib/python3.10/site-packages/conda_package_handling/api.py:29: UserWarning: Install zstandard Python bindings for .conda support
  _warnings.warn("Install zstandard Python bindings for .conda support")
Could not solve for environment specs
The following packages are incompatible
├─ iris-esmf-regrid >=0.7.0  does not exist (perhaps a typo or a missing channel);
├─ iris >=3.6.0  does not exist (perhaps a typo or a missing channel);
├─ prospector >=1.9.0  does not exist (perhaps a typo or a missing channel);
├─ python-stratify >=0.3  does not exist (perhaps a typo or a missing channel);
├─ shapely >=2.0.0  does not exist (perhaps a typo or a missing channel);
└─ sphinx >=6.1.3  does not exist (perhaps a typo or a missing channel).

[zklaus]

Debugging dependency clashes can be a pain. I see the sphinx_rtd_theme popping up. Didn't we switch to the Pydata theme? If I recall correctly, the Rtd theme is not compatible with recent Sphinx versions.

[zklaus]

Hm. My guess on the theme seems to be a red herring, it was there already when the builds still worked and is likely something that Readthedocs is adding. Instead, I think now that this is due to the lack of .conda support. From the log above:

mamba env create --quiet --name 2198 --file environment.yml
/home/docs/.asdf/installs/python/mambaforge-22.9.0-3/lib/python3.10/site-packages/conda_package_streaming/package_streaming.py:19: UserWarning: zstandard could not be imported. Running without .conda support.
  warnings.warn("zstandard could not be imported. Running without .conda support.")
/home/docs/.asdf/installs/python/mambaforge-22.9.0-3/lib/python3.10/site-packages/conda_package_handling/api.py:29: UserWarning: Install zstandard Python bindings for .conda support
  _warnings.warn("Install zstandard Python bindings for .conda support")

This is a problem because .conda is the new format for conda packages (previously .tar.bz2) and all recently built packages are in that format.

[zklaus]

The problem seems to be related to the issue described at https://github.com/conda-forge/zstandard-feedstock/pull/48, i.e. zstandard, the Python binding for the library zstd, depends on internal Zstd API and consequently more tightly on the Zstd version it was built against than was expressed in the requirements. This led to the incompatible combination of zstandard=0.19.0 with zstd=1.5.5 in our env. However, this should not have happened because a repodata patch in https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/484 was used to prevent this. I suspect the problem is that the zstandard package in the environment comes directly from the Mambaforge distribution and didn't get the repodata patch. For now, I hope pinning zstd=1.5.2 will do the trick, but this is severely limiting future upgrades, so a more permanent solution is desirable.