Closed mtraynham closed 3 years ago
I have recreated the issue with a minimal example, https://github.com/mtraynham/poetry_4051
poetry==1.1.5
Creating virtualenv dep-a in /home/foobar/poetry_4051/.venv
Using virtualenv: /home/foobar/poetry_4051/.venv
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 4 installs, 0 updates, 0 removals
• Installing six (1.16.0): Pending...
• Installing six (1.16.0): Installing...
• Installing six (1.16.0)
• Installing grpcio (1.28.1): Pending...
• Installing grpcio (1.28.1): Installing...
• Installing grpcio (1.28.1)
• Installing dep-c (0.1.0 f886f69): Pending...
• Installing dep-c (0.1.0 f886f69): Cloning...
• Installing dep-c (0.1.0 f886f69): Building...
• Installing dep-c (0.1.0 f886f69)
• Installing dep-b (0.1.0 1a00be6): Pending...
• Installing dep-b (0.1.0 1a00be6): Cloning...
• Installing dep-b (0.1.0 1a00be6): Building...
• Installing dep-b (0.1.0 1a00be6)
Installing the current project: dep_a (0.1.0)
- Building package dep-a in editable mode
- Adding dep_a.pth to /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages for /home/foobar/poetry_4051
- Adding the dep_a-0.1.0.dist-info directory to /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages
poetry==1.1.6
Creating virtualenv dep-a in /home/foobar/poetry_4051/.venv
Using virtualenv: /home/foobar/poetry_4051/.venv
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 2 installs, 0 updates, 0 removals, 2 skipped
• Removing grpcio (1.28.1): Pending...
• Removing grpcio (1.28.1): Skipped for the following reason: Not currently installed
• Removing six (1.16.0): Pending...
• Removing six (1.16.0): Skipped for the following reason: Not currently installed
• Installing dep-c (0.1.0 f886f69): Pending...
• Installing dep-c (0.1.0 f886f69): Cloning...
• Installing dep-c (0.1.0 f886f69): Building...
• Installing dep-c (0.1.0 f886f69)
• Installing dep-b (0.1.0 1a00be6): Pending...
• Installing dep-b (0.1.0 1a00be6): Cloning...
• Installing dep-b (0.1.0 1a00be6): Building...
• Installing dep-b (0.1.0 1a00be6)
Installing the current project: dep_a (0.1.0)
- Building package dep-a in editable mode
- Adding dep_a.pth to /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages for /home/foobar/poetry_4051
- Adding the dep_a-0.1.0.dist-info directory to /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages
1.1.6 with diff patch
Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 2 installs, 0 updates, 0 removals, 2 skipped
• Installing six (1.16.0): Pending...
• Installing six (1.16.0): Installing...
• Installing six (1.16.0)
• Installing grpcio (1.28.1): Pending...
• Installing grpcio (1.28.1): Installing...
• Installing grpcio (1.28.1)
• Installing dep-c (0.1.0 f886f69): Pending...
• Installing dep-c (0.1.0 f886f69): Skipped for the following reason: Already installed
• Installing dep-b (0.1.0 1a00be6): Pending...
• Installing dep-b (0.1.0 1a00be6): Skipped for the following reason: Already installed
Installing the current project: dep_a (0.1.0)
- Building package dep-a in editable mode
- Adding dep_a.pth to /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages for /home/foobar/poetry_4051
- Removing existing dep_a-0.1.0.dist-info directory from /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages
- Adding the dep_a-0.1.0.dist-info directory to /home/foobar/poetry_4051/.venv/lib/python3.8/site-packages
We are seeing the same issue. Can confirm adding the whitelist back fixes the issue. We have downgraded to 1.1.5 in the meantime
This looks like the same issue as https://github.com/python-poetry/poetry/issues/3368
It looks like the whitelist
fix referenced in the initial report is indeed the cause; it seems to have been added to prevent the solver from adding newer versions of packages than what are specified in the lockfile. It's not clear to me, but it seems like the bug this was supposed to fix is where a package that pulls in a path or VCS dependency ignores the dependency lock file, resolves potentially newer versions of the transitive dependencies, and adds those to its own lockfile. The whitelist
change prevents the solver from resolving the transitive dependencies, but instead of installing the "locked" packages, it skips them.
In 1.1.5, the solver does resolve the transitive dependencies, but it ignores the versions from the VCS/path deps lockfiles, and installs the latest version that satisfies the dependency specification from the pyproject.toml
file. This is arguably incorrect, but the behavior in 1.1.6 is definitely incorrect.
Not having a firm grasp on the code, and too little time right now to debug in depth, this snippet in particular looks like it might have significance. The deferred dependencies might not need to be resolved, but they should still be loaded?
@sdispater, as the contributor of the breaking PR, what do you think? Surely the transitive package dependencies should all be installed by poetry install
whether the top-level pyproject.toml
specifies the dependencies as path, VCS or otherwise? This is a blocking bug for me and my team, as we rely on path dependencies for organizing our internal tooling.
Likely related issues for 1.1.6: https://github.com/python-poetry/poetry/issues/3956 https://github.com/python-poetry/poetry/issues/4000 https://github.com/python-poetry/poetry/issues/4111
@sdispater I believe #3947 has introduced this issue, but I am not well versed in how or why the lockfile is no longer being used as a whitelist during install, nor why the lockfile generation is somehow different from what an install performs. I wouldn't mind adding a PR to fix the issue, but some guidance would be appreciated; I'm not sure any of us understands the code well enough to suggest a correct fix.
Thanks @mtraynham, I had seen those, I have no idea why I didn't include those issues in my last comment
Also, a correction:
In 1.1.5, the solver does resolve the transitive dependencies, but it ignores the versions from the VCS/path deps lockfiles, and installs the latest version that satisfies the dependency specification from the
pyproject.toml
file. This is arguably incorrect, but the behavior in 1.1.6 is definitely incorrect.
Thinking about this a bit more, if you specify 2 path deps in pyproject.toml
, and those deps have overlapping package dependencies, it's likely that some of the deps in their lock file will have conflicting versions. There's no clear way to prioritize versions from poetry.lock
in path dependencies, so I think the 1.1.5 strategy of ignoring those lock files is actually correct. If someone needs their environment to adhere closely to what's specified in the path dep lockfiles, they should use a manually created venv and install the path deps by hand, in the order they want to prioritize the poetry.lock
preferences for each dep.
The root cause has been identified (see #4202) and this will be fixed in the next bugfix release.
Note that you will to regenerate the lock file for the fix to take effect.
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
[X] I am on the latest Poetry version.
[X] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (
-vvv
option).OS version and name: Linux/Mac
Poetry version: 1.1.6
Link of a Gist with the contents of your pyproject.toml file: https://github.com/mtraynham/poetry_4051
Issue
There seems to be a regression with transitive dependencies of a Git dependency. I haven't exactly figured out why, but the change for the whitelist on the installer is what causes the transitive dependency to be flagged for removal rather than keeping it. The dependency is in the lock file, but when it goes to install the project, it skips it.
This particular pull causes the issue, https://github.com/python-poetry/poetry/pull/3947. Reverting the whitelist changes seems to correct the issue: https://github.com/python-poetry/poetry/pull/3947/files#diff-95c2e34175a95676a55830f8442a68da156ae6a31dfec68317a2fd746be2243fL292-L297 https://github.com/python-poetry/poetry/pull/3947/files#diff-95c2e34175a95676a55830f8442a68da156ae6a31dfec68317a2fd746be2243fL308
This may actually be caused because I have 1 git dependency, which depends on another git dependency. E.g. the dependency tree looks like:
Project A -> Project B (git; poetry) -> Project C (git; setup.py) -> PyPi dep (not installed, but in the lock file)