Poetry.lock module versions appear a little out of date

[fseesink]

While tinkering around, ran into the following which has been the case for awhile now. If I am in a virtualenv where I have Nornir installed using the usual pip install nornir, and I do a pip list --outdated, then try to do a pip install --upgrade <module> for certain modules, inevitably it fails, always due to the dependencies in Nornir.

Currently this includes the ruamel.yaml and typing-extensions modules, which are at v0.17.20 and v4.0.1, respectively, as I type this. Unfortunately, on my system they are stuck at 0.16.3 and 3.10.0.2, as I installed Nornir awhile ago. Took me a little bit to find the issue, as you all have a poetry.lock file (meaning you use Poetry for your package management) but not a basic pip requirements.txt file in your repo. But eventually found what I was after. So at least now I've been able to get things patched closer by doing a pip install --upgrade "ruamel.yaml==0.17.17".

While both modules have seen updates, though, it seems Nornir's poetry.lock file does not reflect this. I realize it's challenging to maintain this, but might it be worth considering to adjust the requirements to be a little more flexible? That is, instead of setting specific versions, allow some leeway. For ruamel.yaml, possibly set the version to 0.17.* instead of the specific 0.17.17. this way small patches are allowed within the same subversion. Admittedly for typing-extensions, they went up a full version. So that one likely requires testing.

But as we are witnessing more and more, when a program relies on other modules/packages/libraries, it is just as important to make sure you allow for those to be patched as well as your own code itself. Anyway, just something to consider.

[dbarrosop]

Hello, thanks for bringing this issue, let me start with a few clarifications:

1. Pip doesn't use poetry.lock. Pip uses what's defined in pyproject.toml (kind of, it uses what's on PKG-INFO which is populated by poetry during the build with what's on pyproject.toml). In this case we have the following dependencies:

"ruamel.yaml" = ">=0.16"
mypy_extensions = "^0.4.1"
typing_extensions = "^3.7"

And I can see pip does the right thing:

➜  nornir git:(develop) python3 -m venv /tmp/deleteme
➜  nornir git:(develop) . /tmp/deleteme/bin/activate
(deleteme) ➜  nornir git:(develop) pip install -U nornir
Collecting nornir
  Downloading nornir-3.2.0-py3-none-any.whl (30 kB)
Collecting importlib-metadata<5,>=4
  Downloading importlib_metadata-4.10.0-py3-none-any.whl (17 kB)
Collecting ruamel.yaml>=0.16
  Downloading ruamel.yaml-0.17.20-py3-none-any.whl (109 kB)
     |████████████████████████████████| 109 kB 15.3 MB/s
Collecting typing_extensions<4.0,>=3.7
  Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting mypy_extensions<0.5.0,>=0.4.1
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting zipp>=0.5
  Downloading zipp-3.7.0-py3-none-any.whl (5.3 kB)
Collecting ruamel.yaml.clib>=0.2.6
  Downloading ruamel.yaml.clib-0.2.6.tar.gz (180 kB)
     |████████████████████████████████| 180 kB 63.3 MB/s
Using legacy 'setup.py install' for ruamel.yaml.clib, since package 'wheel' is not installed.
Installing collected packages: zipp, ruamel.yaml.clib, typing-extensions, ruamel.yaml, mypy-extensions, importlib-metadata, nornir
    Running setup.py install for ruamel.yaml.clib ... done
Successfully installed importlib-metadata-4.10.0 mypy-extensions-0.4.3 nornir-3.2.0 ruamel.yaml-0.17.20 ruamel.yaml.clib-0.2.6 typing-extensions-3.10.0.2 zipp-3.7.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/tmp/deleteme/bin/python3 -m pip install --upgrade pip' command.
(deleteme) ➜  nornir git:(develop) pip freeze
importlib-metadata==4.10.0
mypy-extensions==0.4.3
nornir==3.2.0
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
typing-extensions==3.10.0.2
zipp==3.7.0

a. It's getting latest ruaml.yaml as per the rule >=0.16 b. It's getting latest 3.x version of typing extensions as per the rule ^3.7

2. poetry.lock is only used by poetry itself, which is meant for development purposes. This is useful to separate updating deps from changing code which, in turn, is useful for reproducible builds. You don't want to code locally, push to github and have things fail randomly because the environment there is different from yours.

Now, to your issue at hand, as you can see from my test the issue you are having with ruaml.yaml must be something else as pip is picking the latest available. Regarding typing-extensions, they went through a significant enough change that they bumped the version from 3.x to 4.x so what needs to be done here is open a PR, change the rule to ^4 instead of ^3.7 (which is short for >=4.0.0,<5.0.0 and >=3.7.0,<4.0.0 respectively), and test to make sure nothing breaks.

Being too permissive means that what works on my machine when developing might not necessarily work on your machine when you install it. Being too little permissive means outdated dependencies, which isn't good either so we need to strike a balance. Given the rules we have defined now, if you think there is some improvement to be done, don't hesitate to let me know.

You are also welcomed to open a PR to bump typing_extensions and/or mypy_extensions.

[fseesink]

That is so weird. I've been dealing with this around ruamel.yaml for some time now. That is, I have a virtualenv setup for a local project and installed Nornir awhile back. And ruamel.yaml was at 0.16.3 then. But every time I would get back into that venv and do a pip list --outdated, I'd see among the module updates that ruamel.yaml had one. But every single time I tried to update it using pip install --upgrade ruamel.yaml, it would kick back an error due to a dependency with Nornir. So I'd have to revert using pip install --upgrade "ruamel.yaml==0.16.3" and then pip was happy.

Now yesterday as I wrote, after digging into the files I eventually did a pip install --upgrade "ruamel.yaml=0.17.7" and that worked fine. And then after reading your reply, I went looking by creating a new venv for testing, and sure enough doing a clean pip install nornir brings down ruamel.yaml 0.7.20. So was like, "What the?"

So went back to my dev venv and tried the upgrade again. First I thought to try simply doing pip install nornir. And as expected, since I already had Nornir 3.2.0 installed, it showed all the dependencies were already installed. Then did the upgrade of ruamel.yaml as mentioned before. And son of a... it worked! So now it's all patched up on that front around ruamel.yaml. So 'yay!' to that. 😄

As for the whole pyproject.toml bit, that's my stupidity, sorry. We've mostly been doing rather basic work in Python, using a requirements.txt file. Need to really sit down and see where this fits into things. So thanks for that. If nothing else, I'm gonna get to learn something new. And as for Poetry, ran across it awhile back, but like a lot of things in development, you kind of get used to certain ways/tools/etc. of doing things, and simply haven't had time to see what that brings to the table and if it's worth it. My last go round I was tinkering with pip vs. pipenv, etc. And for my own workflow, as I work between 3 computers where I sync certain folders b/w them so I can move effortlessly, using stock pip worked better. pipenv stores the environs outside the dev directory structure under ~/, which under other circumstances might be nicer/cleaner. But in my case, it'd mean I had to remember to repeat my pip installs/upgrades on each machine. But as I'm setup now, I do it on one, and everything just syncs to the other machines. But I digress.

Back to what was biting me. Best I can guess is something wasn't updated properly when I did the update of Nornir itself (i.e., pip install --upgrade nornir), leaving behind the older requirements. Then when I did pip install nornir, possibly things got updated. Sorry, I'm guessing right now. But I say this because I went and built a clean venv, installed Nornir 3.1.1 specifically, then tried to update ruamel.yaml as before. And sure enough, I got the error I was used to seeing. That is,

% ls -al
total 0
drwxr-xr-x   2 frank  staff   64 Jan  7 11:05 .
drwx------@ 22 frank  staff  704 Jan  7 11:01 ..
% virtualenv venv
created virtual environment CPython3.10.1.final.0-64 in 393ms
  creator CPython3Posix(dest=/Users/frank/Desktop/test/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/frank/Library/Application Support/virtualenv)
    added seed packages: pip==21.3.1, setuptools==60.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
% source venv/bin/activate
(venv) % pip list
Package    Version
---------- -------
pip        21.3.1
setuptools 60.2.0
wheel      0.37.1
(venv) % pip install "nornir==3.1.1"
Collecting nornir==3.1.1
  Using cached nornir-3.1.1-py3-none-any.whl (30 kB)
Collecting mypy_extensions<0.5.0,>=0.4.1
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting typing_extensions<4.0,>=3.7
  Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting ruamel.yaml<0.17,>=0.16
  Using cached ruamel.yaml-0.16.13-py2.py3-none-any.whl (111 kB)
Installing collected packages: typing-extensions, ruamel.yaml, mypy-extensions, nornir
Successfully installed mypy-extensions-0.4.3 nornir-3.1.1 ruamel.yaml-0.16.13 typing-extensions-3.10.0.2
(venv) % pip list
Package           Version
----------------- --------
mypy-extensions   0.4.3
nornir            3.1.1
pip               21.3.1
ruamel.yaml       0.16.13
setuptools        60.2.0
typing-extensions 3.10.0.2
wheel             0.37.1
(venv) % pip list --outdated
Package           Version  Latest  Type
----------------- -------- ------- -----
nornir            3.1.1    3.2.0   wheel
ruamel.yaml       0.16.13  0.17.20 wheel
setuptools        60.2.0   60.3.1  wheel
typing-extensions 3.10.0.2 4.0.1   wheel
(venv) % pip install --upgrade ruamel.yaml
Requirement already satisfied: ruamel.yaml in ./venv/lib/python3.10/site-packages (0.16.13)
Collecting ruamel.yaml
  Using cached ruamel.yaml-0.17.20-py3-none-any.whl (109 kB)
Collecting ruamel.yaml.clib>=0.2.6
  Using cached ruamel.yaml.clib-0.2.6-cp310-cp310-macosx_10_9_universal2.whl (149 kB)
Installing collected packages: ruamel.yaml.clib, ruamel.yaml
  Attempting uninstall: ruamel.yaml
    Found existing installation: ruamel.yaml 0.16.13
    Uninstalling ruamel.yaml-0.16.13:
      Successfully uninstalled ruamel.yaml-0.16.13
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nornir 3.1.1 requires ruamel.yaml<0.17,>=0.16, but you have ruamel-yaml 0.17.20 which is incompatible.
Successfully installed ruamel.yaml-0.17.20 ruamel.yaml.clib-0.2.6
(venv) % pip install --upgrade nornir
Requirement already satisfied: nornir in ./venv/lib/python3.10/site-packages (3.1.1)
Collecting nornir
  Using cached nornir-3.2.0-py3-none-any.whl (30 kB)
Requirement already satisfied: mypy_extensions<0.5.0,>=0.4.1 in ./venv/lib/python3.10/site-packages (from nornir) (0.4.3)
Requirement already satisfied: ruamel.yaml>=0.16 in ./venv/lib/python3.10/site-packages (from nornir) (0.17.20)
Requirement already satisfied: typing_extensions<4.0,>=3.7 in ./venv/lib/python3.10/site-packages (from nornir) (3.10.0.2)
Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in ./venv/lib/python3.10/site-packages (from ruamel.yaml>=0.16->nornir) (0.2.6)
Installing collected packages: nornir
  Attempting uninstall: nornir
    Found existing installation: nornir 3.1.1
    Uninstalling nornir-3.1.1:
      Successfully uninstalled nornir-3.1.1
Successfully installed nornir-3.2.0
(venv) % pip install --upgrade ruamel.yaml
Requirement already satisfied: ruamel.yaml in ./venv/lib/python3.10/site-packages (0.17.20)
Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in ./venv/lib/python3.10/site-packages (from ruamel.yaml) (0.2.6)

Sorry, the syntax color highlighting is missing here, but hopefully the ERROR: lines are pretty obvious. Those show up in red in my shell so kind of get your attention.

But as you can see here, after that I upgraded Nornir itself, then ruamel.yaml, and darned if it didn't update just fine! So... yeah, I give up.

Anyway, thanks for the help all the same, @dbarrosop. At least now my setup is patched properly. And yes, I get it about full version changes like with typing-extensions. That's a different beast. (We have to deal with that in our project as well, as we need to upgrade the Django LTS release we are using soon.)

[dbarrosop]

Nornir 3.1.1 had ruaml.yaml pinned to a specific version so the error is expected. That was corrected in nornir 3.2.0.

In any case, I'd recommend treating virtualenvs as "immutable", meaning, don't bother upgrading/downgrading packages in existing virtualenvs; just delete it, recreate it and install everything from scratch. Seems overkill but it takes very little time/effort to do it and you may save yourself a lot of trouble dealing with pip and botched upgrades/downgrades (like this particular issue :P)

[dbarrosop]

I am closing this issue as I think there is nothing to do here but feel free to reopen otherwise.