less strict version requirements?

[mathause]

Pyam has relatively strict version pins - this makes it difficult to install alongside other packages (or in an existing environment). Would you consider relaxing these requirements?

Many packages in the scientific stack follow NEP-29 or SPEC-00 (which are very similar). I ran your requirements through xarrays interpretation of the rules, which suggests that your requirements for python, numpy, pandas and xlsxwriter should be relaxed.

Package           Required             Policy               Status
----------------- -------------------- -------------------- ------
python            3.10    (2021-10-04) 3.9     (2020-10-05) > (!)
iam-units         2020.4  (2020-06-07) 2022.10 (2022-10-27) <
ixmp4             0.8     (2024-03-20) 0.8     (2024-03-20) =
matplotlib        3.6     (2022-09-16) 3.7     (2023-02-14) <
numpy             1.26    (2023-09-17) 1.23    (2022-06-23) > (!)
openpyxl          3.1     (2023-01-31) 3.1     (2023-01-31) =
pandas            2.1     (2023-08-30) 1.5     (2022-09-19) > (!)
Pint              0.13    (2020-06-25) 0.20    (2022-10-25) <
PyYAML            6.0     (2021-10-16) 6.0     (2021-10-16) =
scipy             1.10    (2023-01-04) 1.10    (2023-01-04) =
seaborn           0.11    (2020-09-08) 0.12    (2022-09-06) <
six               1.16    (2021-05-05) 1.16    (2021-05-05) =
wquantiles        0.6     (2021-05-26) 0.6     (2021-05-26) =
xlsxwriter        3.1     (2023-04-27) 3.0     (2021-08-10) > (!)

(NOTE: you require xlswriter 3.2, but this is not yet available on conda-forge, https://github.com/conda-forge/xlsxwriter-feedstock/pulls)

edit: added enviroment.yml file I used to run min_deps_check.py

```yaml name: pyam-tests channels: - conda-forge - nodefaults dependencies: - python=3.10 - iam-units=2020.4 - ixmp4=0.8 - matplotlib=3.6 - numpy=1.26 - openpyxl=3.1 - pandas=2.1 - Pint=0.13 - PyYAML=6.0 - scipy=1.10 - seaborn=0.11 - six=1.16 - wquantiles=0.6 - xlsxwriter=3.1 ```

[gidden]

Thanks @mathause. I have heard from others that the strict requirements of pyam are a pain point as well and would be in favor for reducing them as much as possible.

@phackstock could you chime in here? I know we had pretty strict requirements at one point due to some plotting features, but that should be covered now simply by testing environments.

I suspect among all of these, the pandas requirement will likely be a blocker for us, as I believe we transitioned everything to 2.0. @danielhuppmann can confirm that.

[danielhuppmann]

Thanks for the note about the xlsxwriter dependency being very strict and not yet released on condo - see #841 for a fix, seems that was indeed too restrictive.

The other highlighted dependencies are inherited from the brand-new ixmp4 package (read the docs), which will form an integral component of the IIASA Scenario Services software stack. Here, we wanted to make use of the solution offering the best performance (ie., modern pandas) and good documentation, in particular good type hints (ie py3.10).

@mathause, is there a particular use case for which you are using pyam and where the current requirements are too restrictive for you?

[mathause]

I tried to install pyam using conda in an existing environment which is about a year old (python 3.11; numpy 1.25; pandas 2.0; matplotlib 3.7.2). It wanted to install pyam v1.6 - for the wrong reason. pyam 1.7 is where you added an upper pin for numpy (see https://github.com/conda-forge/pyam-feedstock/pull/47/files).

For newer environments it's a less of a problem. However, on conda-forge/pyam-feedstock you have several upper pins, which make installing difficult. The upper pins probably come from poetry, which I think has some sub-optimal recommendations. I find myself agreeing with this approach instead: https://www.youtube.com/watch?v=WSVFw-3ssXM

The other highlighted dependencies are inherited from the brand-new ixmp4 package (read the docs), which will form an integral component of the IIASA Scenario Services software stack. Here, we wanted to make use of the solution offering the best performance (ie., modern pandas) and good documentation, in particular good type hints (ie py3.10).

Yes, I see no problems violating NEP29 for good reasons and don't ask to put in work to support old versions. Also I think the upper pins are more problematic than the lower ones.

Considering ixmp4 - I also see poetry style pins... So again maybe these could be loosened? If these libraries are meant to be installed stand-alone in an environment, this all does not matter. If they should be installed alongside other packages (e.g. be part of a data analysis pipeline) the dependency pins should be made less strict.

---

So my personal take would be (1) align the dependencies on conda-forge and remove the upper pins, (2) set up tests for the oldest supported and upstream dev versions (not sure if you already do this); (3) be less 'aggressive' about bumping requirements in the future.

[danielhuppmann]

Indeed, we already test for the oldest explicitly supported versions of the main dependencies to guard against new features not working as expected on older setups, see the pytest-legacy workflow.

And yes, we switched to poetry recently, and we'll keep an eye on removing upper pins going forward.

And this behavior by conda of trying to guess which version to install based on some non-obvious constraints in a given setup is why I personally moved away from using conda.

[danielhuppmann]

@mathause, we took your concern to heart and have now also added tests to ixmp4 that this package supports all versions since 2.0.0 (see https://github.com/iiasa/ixmp4/pull/80) and pyam supports all versions since 2.1.2. We have also removed upper bounds in poetry and the conda releases.

Closing as no further direct action is planned, please re-open or start a new issue if you have specific concerns.

[mathause]

Thanks a lot - I appreciate this! I hope this was not too much work. I am waiting for the updated conda package and will install it then.