pytest -m "not installation" fails: `pytest-metadata` not in line with new `pytest==8.0.0` and reverts to older pytest-metadata=2.0.0

[mwjury]

I have installed ESMValTool and ESMValCore from source (both v2.10.x; but this also happens for the main branches) as outlined, using an up to date mamba (Mambaforge-Linux-x86_64.sh).

The installation works fine, however both pytest -m "not installation" and pytest -n 4 yield:

/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pytest_metadata/plugin.py:123: PytestUnknownMarkWarning: Unknown pytest.mark.optionalhook - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
  @pytest.mark.optionalhook
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pytest_html/basereport.py", line 73, in _generate_environment
INTERNALERROR>     from pytest_metadata.plugin import metadata_key
INTERNALERROR> ImportError: cannot import name 'metadata_key' from 'pytest_metadata.plugin' (/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pytest_metadata/plugin.py)
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pytest_html/basereport.py", line 167, in pytest_sessionstart
INTERNALERROR>     self._report.set_data("environment", self._generate_environment())
INTERNALERROR>                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/nas/home/maj/mambaforge/envs/evt210/lib/python3.11/site-packages/pytest_html/basereport.py", line 78, in _generate_environment
INTERNALERROR>     metadata = self._config._metadata
INTERNALERROR>                ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> AttributeError: 'Config' object has no attribute '_metadata'

Additionally, and probably connected. Running any recipes gives: ImportError: The ESMFMKFILE environment variable is not available.

python -c 'import esmvalcore; print(esmvalcore.__path__[0])' gives the correct path

Could anyone give me any leads on how to overcome this?

[valeriupredoi]

thanks for raising this @mwjury - I saw this behaviour yesterday, immediately after we started picking up the brand new pytest=8.0.0 which is a rather buggy major release of pytest, I'm onto it :+1:

[valeriupredoi]

Re:

Additionally, and probably connected. Running any recipes gives: ImportError: The ESMFMKFILE environment variable is not available.

this is not connected to the pytest issue, this is due to xesmf now dealing with environment variables via an init file, ESMFMKFILE. This is normally picked up and placed in the runtime environment automatically via a conda activate <environment name> if the package is installed from conda-forge - so make sure xesmf is from conda forge, as it should be. If you are doing a module-based environment activation (ie not via conda activate) or any other path-based environment setup, you can set the env variable automatically eg with a setenv("ESMFMKFILE","/apps/jasmin/community/esmvaltool/miniconda3_py310_23.1.0_2/envs/esmvaltool/lib/esmf.mk") - this is from the central install on JASMIN where I set up the ESMFMKFILE variable to point to where that lives in the miniconda3 dir - you'll have to change the path to your file, but it always lives in envs/esmvaltool/lib/esmf.mk

[valeriupredoi]

here's the issue with pytest: https://github.com/conda-forge/pytest-feedstock/issues/170

[mwjury]

Thanks for the fast reply @valeriupredoi !

pytest is 7.4.4 though.

pytest                    7.4.4                    pypi_0    pypi
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-env                1.1.3              pyhd8ed1ab_0    conda-forge
pytest-html               4.1.1              pyhd8ed1ab_0    conda-forge
pytest-metadata           2.0.0              pyhd8ed1ab_0    conda-forge
pytest-mock               3.12.0             pyhd8ed1ab_0    conda-forge
pytest-mypy               0.10.3                   pypi_0    pypi
pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge

I am a bit puzzled by the xesmf thing. When activating the environment conda prompts all the etc/conda/activate.d/*.sh scripts, but does not update any of the paths. When updating the paths in python with os.environ.update({"ESMFMKFILE": "/nas/home/maj/mambaforge/envs/evt210/lib/esmf.mk"}) esmpy import etc works. Did this ever happen to you by any chance?

[valeriupredoi]

yes but your pytest-metadata is ancient (2.0.0) - you need to upgrade that plugin to 3.0.0 (that is exactly the issue with pytest 8.0 - it downgrades the plugin to 2.0.0). Your deps are a little mixed though, too - you shouldn't have any installed from PyPI (all need be from conda-forge). At any rate, the xesmf issue is well known to us and people have hit all manners of troubles with it (eg @ehogan for the MetO central installation, myself for JASMIN, us collectively for readthedocs deployments etc) - ESMPy had this introduced in 8.4.0 (I believe) and ever since we have to look twice if that init file is there in the conda env - as for your current issue, I'd recommend completely recreating the environment, activating it, and reinstalling the Tool; I'd also strongly recommend updating both conda and mamba to be modern, that will 100% surely sort it out for you

[valeriupredoi]

I fixed the plugin so all should work well when the new plugin build gets on conda-forge (a few hours now) and gets pulled in our env: https://github.com/conda-forge/pytest-metadata-feedstock/pull/15 so closing this. @mwjury do please open another issue about xesmf/ESMPy if a new env and installation are still problematic for you :+1: :beer:

[mwjury]

Thanks @valeriupredoi ; pytest works now. It currently shows two FAILURES connected to pytest.PytestRemovedIn8Warning: Passing None has been deprecated.

I still did not solve the conda issue (recreated it several times..). In the end all scripts in etc/conda/activate.d upon activation get called but are somehow not sourced. I haven't found a solution as of yet, other than sourcing them 'manually'.

[mwjury]

Solved it, stupidity from my side. 🍺

[valeriupredoi]

@mwjury those two tests are the MLR tests that I opened an issue for yesterday - indeed we'll fix them, I'll plug a fix if @schlunma doesn't have time :beer:

About the ESMFMKFILE woes - great! It'd be worth posting a step by step of your operations see what needed be done at your end, and maybe highlighting that for users in the docs (if you remember the steps, and have time to post them, that is). Cheers!

[valeriupredoi]

Manu fixed those two tests in https://github.com/ESMValGroup/ESMValTool/pull/3517 which is now merged :+1: