python dependency version conflict using attrs between r2d2 v3 web server and eccodes 1.5.0

[ericlingerfelt]

Describe the bug

attrs==23.2.0 is listed as required for the r2d2 server components. attrs==21.4.0 (as the py-attrs/21.4.0 module) is required by py-eccodes/1.5.0 as part of /Users/eric2/spack-stack-1.7.0/envs/skylab-7.0.0/install/apple-clang/15.0.0/py-attrs-21.4.0-i47jwy7/lib/python3.10/site-packages, for example. If py-attrs/21.4.0 is loaded, the following error occurs and the local r2d2 server will not start.

(venv) JCSDAeric2-MBP:server eric2$ module load  py-attrs/21.4.0
(venv) JCSDAeric2-MBP:server eric2$ python3 app.py
Traceback (most recent call last):
  File "/Users/eric2/jedi/jedi-bundle/r2d2/server/app.py", line 5, in <module>
    from flasgger import Swagger, swag_from
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/flasgger/__init__.py", line 9, in <module>
    from jsonschema import ValidationError  # noqa
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/jsonschema/__init__.py", line 13, in <module>
    from jsonschema._format import FormatChecker
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/jsonschema/_format.py", line 11, in <module>
    from jsonschema.exceptions import FormatError
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/jsonschema/exceptions.py", line 15, in <module>
    from referencing.exceptions import Unresolvable as _Unresolvable
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/referencing/__init__.py", line 4, in <module>
    from referencing._core import Anchor, Registry, Resource, Specification
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/referencing/_core.py", line 87, in <module>
    class Specification(Generic[D]):
  File "/Users/eric2/jedi/venv/lib/python3.10/site-packages/referencing/_core.py", line 113, in Specification
    ] = field(alias="anchors_in")
TypeError: field() got an unexpected keyword argument 'alias'

If py-attrs/21.4.0 is unloaded and the pip-installed attrs==23.2.0 is used, the following successful result occurs and the r2d2 server will start.

(venv) JCSDAeric2-MBP:server eric2$ module unload py-attrs
Lmod Warning: 
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: py-attrs/21.4.0 (required by: py-eccodes/1.5.0)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(venv) JCSDAeric2-MBP:server eric2$ python3 app.py
 * Serving Flask app 'app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000/
 * Running on http://10.0.0.105:5000/
Press CTRL+C to quit

To Reproduce See above.

Expected behavior This conflict should not occur and the server will start using the correct library.

System: MacOS Sonoma with Python 3.10.13 from spack-stack.

[eap]

Is there any reason we need to be using Python from spack-stack? This feels like an opportunity to divorce the r2d2 core implementation from skylab dependencies since they will be running on separate platforms. The r2d2 client can continue being coupled to skylab dependencies.

As an interim solution, the r2d2 library can continue to use spack-stack dependencies and all the version-conflicting dependencies will go into the server core, but once we migrate to full client/server setup it won't matter.

@ericlingerfelt, Thoughts?

[ericlingerfelt]

Hi @eap, In my Skylab working environment, which includes R2D2 and EWOK, I always utilize the spack-stack python for the jedi-bundle build and my virtual environment. My installation of R2D2 and EWOK are inside the jedi-bundle directory. I only do this since that is what has always worked and that is what is recommended by our documentation. The purpose of spack-stack is to provide these python dependencies consistently across platforms for all users.

For now, users can simply unload the py-attrs/21.4.0 module.

[climbfuji]

Let's be clear that r2d2-client will be using spack-stack and there are many reasons why Python and packages are built by and provided by spack. It's in fact one of the main benefits of spack that it provides a single, consistent environment for compiled packages, python packages, and hybrid packages.

The r2d2 server implementation will almost certainly not be using spack - it's a simple setup that users do not need to know anything about, just the INFRA admins.

[ericlingerfelt]

Sounds good, @eap and @climbfuji. Thanks!