Make `prospector` an optional requirement

[valeriupredoi]

Right now the only thing between us and supporting Python=3.11 (speaking purely from an env blocker point of view, we still have issues since iris-esmf-regrid installs with 3.11 but don't work with it) is prospector and one of its dependencies, see #1832 - I reckon we ought to make prospector an optional dependency since we're not importing it anywhere, we just recommend users make use of it - pls, we've had all manners of issues with it in the past - what do you think @ESMValGroup/technical-lead-development-team

[bouweandela]

Since it is only installed if you use the [develop] extra, it is already optional: https://github.com/ESMValGroup/ESMValCore/blob/b1e5fece2a10b22b1c84e66dd9a4a906d3c81e83/setup.py#L93-L101

[valeriupredoi]

not in the conda environment yml it's not, bud - that was my point, remove it from there

[bouweandela]

I tried to test this, but it didn't work for me. After setting python=3.11 and removing prospector from environment.yml, I get the following error:

$ mamba env create -f environment.yml -n test
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Looking for: ['cartopy', 'cf-units', 'cftime', 'dask', 'compilers', "esgf-pyclient[version='>=0.3.1']", "esmpy[version='!=8.1.0,<8.4']", 'filelock', 'fiona', 'fire', 'geopy', 'humanfriendly', 'importlib_resources', "iris[version='>=3.4.0']", 'iris-esmf-regrid', 'isodate', 'jinja2', 'nc-time-axis', 'nested-lookup', 'netcdf4', 'numpy', 'packaging', 'pandas', 'pillow', 'pip!=21.3', 'prov', 'psutil', 'py-cordex', 'pybtex', 'python=3.11', 'python-stratify', 'pyyaml', 'requests', "scipy[version='>=1.6']", 'shapely', 'yamale', "autodocsumm[version='>=0.2.2']", 'ipython', 'nbsphinx', "sphinx[version='>=5']", 'sphinx_rtd_theme', 'flake8', "mypy[version='>=0.990']", "pytest[version='>=3.9,!=6.0.0rc1,!=6.0.0']", "pytest-cov[version='>=2.10.1']", 'pytest-env', 'pytest-html!=2.1.0', "pytest-metadata[version='>=1.5.1']", 'pytest-mypy', 'pytest-mock', 'pytest-xdist', 'types-requests', 'types-pkg_resources', 'types-pyyaml', 'codespell', 'docformatter', 'isort', 'pre-commit', 'yamllint', 'yapf']

Encountered problems while solving:
  - package iris-3.4.1-pyhd8ed1ab_0 requires cartopy >=0.21, but none of the providers can be installed

so it looks like there are still more open issues. My environment.yml file looks like this:

$ cat environment.yml
---
name: esmvaltool
channels:
  - conda-forge
  - nodefaults

dependencies:
  - cartopy
  - cf-units
  - cftime
  - dask
  - compilers
  - esgf-pyclient>=0.3.1
  - esmpy!=8.1.0,<8.4  # see github.com/ESMValGroup/ESMValCore/issues/1208
  - filelock
  - fiona
  - fire
  - geopy
  - humanfriendly
  - importlib_resources
  - iris>=3.4.0
  - iris-esmf-regrid
  - isodate
  - jinja2
  - nc-time-axis
  - nested-lookup
  - netcdf4
  - numpy
  - packaging
  - pandas
  - pillow
  - pip!=21.3
  - prov
  - psutil
  - py-cordex
  - pybtex
  - python=3.11
  - python-stratify
  - pyyaml
  - requests
  - scipy>=1.6
  - shapely
  - yamale
  # Python packages needed for building docs
  - autodocsumm>=0.2.2
  - ipython
  - nbsphinx
  - sphinx>=5
  - sphinx_rtd_theme
  # Python packages needed for testing
  - flake8
  - mypy>=0.990
  - pytest>=3.9,!=6.0.0rc1,!=6.0.0
  - pytest-cov>=2.10.1
  - pytest-env
  - pytest-html!=2.1.0
  - pytest-metadata>=1.5.1
  - pytest-mypy
  - pytest-mock
  - pytest-xdist
  # Not on conda-forge - ESMValTool_sample_data==0.0.3
  # Still for testing, MyPy library stubs
  - types-requests
  - types-pkg_resources
  - types-PyYAML
  # Python packages needed for installing in development mode
  - codespell
  - docformatter
  - isort
  - pre-commit
  #- prospector!=1.1.6.3,!=1.1.6.4
  # Not on conda forge - vprof
  - yamllint
  - yapf

[valeriupredoi]

@bouweandela I like your hands-on approach but #1832 is the place for you to go :grin: esmpy pin is your enemy here - look at the tests in that PR and you'll see what I am about

[valeriupredoi]

like I says above, two outstanding issues prevent us from supporting Python=3.11:

• prospector in the conda environment via conda env file
• and iris-esmf-regrid not ready for new esmpy but we have a PR for that https://github.com/SciTools-incubator/iris-esmf-regrid/pull/240

[bouweandela]

Let's wait until prospector is actually the last issue until deciding anything on it.

[valeriupredoi]

oh sure, this is long range - will come back to it after iris-esmf-regrid gets the ducks in a 3.11 line, I finally opened it coz it's been on my mind for a while but I kept forgetting about it :+1:

[valeriupredoi]

we fixed prospector, and now I am a conda feedstock maintainer of it too (I know, what accolades :laughing: ) so it now works with Python 3.11 and we'll make it work with stuff in the future, should we need it to