Ability to override/ignore sub-dependencies

[zehauser]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

(However, it is related to https://github.com/sdispater/poetry/issues/436).

Issue

In the dark, old world of Python packaging, sub-dependencies are handled very poorly. If I recall correctly, pip will happily install a sub-dependency despite conflicting versions being specified by two direct dependencies... in fact I think which version it ends up installing depends on the order in requirements.txt. Yuck! Only very recently has it even started issuing a warning for cases like this.

In contrast, poetry does this right. It computes the entire dependency tree and will complain if there are conflicts anywhere in the tree.

But... many packages out there are not specifying their dependencies properly. Even if they are, there's always the possibility that their specified dependencies are a tighter range than they strictly need to be.

Is there a way to tell Poetry to force a specific version (or version) range of a dependency in cases like this — or in other words, to ignore a dependency specification of another dependency somewhere in the tree? If not, should there be?

[swarmer]

+1 Looks like this can be enough of a problem to make poetry unusable for many non-library projects, so this is quite important.

[zehauser]

@sdispater You must be super busy, but what are your initial thoughts on this?

[sdispater]

I don't think this is desirable. This would require a lot of work and add complexity to the resolver - which is already complex due to the Python specific ecosystem - because of some packages specifying their dependencies poorly. I don't want Poetry to have to make up for a lack of proper tools or proper specifications, there is already a lot of work to be done as it is.

This is the job of each package's maintainers to ensure their dependencies are correct and loose enough to not create conflict.

If we ever want to have an ecosystem similar to what other languages already have, we have to draw the line somewhere and enforce everyone to contribute to the common goal. Poetry helps with that by making it easier to build and manages Python projects.

[danielkza]

@sdispater While you are completely correct in principle, in reality there are cases where not being able to perform overrides means being completely unable to install some packages, specially when multiple projects specify conflicting version constraints.

Waiting for the whole ecosystem to improve will take years, and in the meanwhile Poetry is unusable in some cases, which is a shame considering it brings lots of improvements in other areas.

As a practical example, awscli requires a particular version range colorama while docker-compose requires another. That makes it completely impossible to install both simultaneously, and neither of the projects are willing to make a change. But in practice both programs work correctly (since I've been using them in the same venv for years).

Even other package managers that work with more "well-behaved" ecosystems allow overrides e.g. Maven, SBT and others in Java world.

[fsworld009]

Would also like to have this option, especially as of now there are packages cannot be installed because poetry fails to parse their setup.py, like #904.

[drunkwcodes]

Uncompromising claim is good to formalize the package versioning.

But the conflicts are existing for sure. And packages are ofter more useful than it claims. An error mitigation strategy is a must.

I think it's good to have a export option to write those conflicts in a requirements.txt. And we can review & edit them like dealing a merge conflict.

So the work flow will be like:

1. poetry export --ignore-conflicts
2. Resolve conflicts in requirements.txt.
3. pip install -r requirements.txt

This can be a plug-in if necessary. Maybe related to #558 #663 & #955.

[RXminuS]

I've ran into exactly the same issue as @danielkza. But I don't think we need to make it that complicated, if you could just tell poetry to completely ignore the dependencies specified by a package then you can always just specify your own.

Also at the moment, it can take a really long time for poetry install to try every combination of dependencies, there must be a better way than iterating down the version numbers?

[floer32]

I am sympathetic to the points about having some way to override or accelerate or skip resolution on an opt-in basis, but also always want to avoid making compromises to the "real, thorough" way of doing a task like this for the sake of proactively optimizing some efficient way...

Explicit control seems best... So you can do real/thorough at maintenance times, in master CI/CD or base docker/VM image baking, etc etc, then more efficient when there has not been a change in the reqs. (And that 'has there been a change?' part is well-solved by things like git-checks and/or docker image caching, and I do think it's best to keep that burden on those tools, and really avoid complicating concepts of cache,state, or history in Poetry itself.)

---

I like @drunkwcodes 's workflow suggestion, and agree that #558 seems related. It really seems there is another subcommand or workflow waiting to be fleshed out...

As @drunkwcodes says maybe it could be a plugin. Personally I am fine "dropping down" to pip for these cases, and then continuing to rely on Poetry as the "master" and what I rely on for longevity and maintenance.

[floer32]

brainstorming what this new subcommand or plugin may look like

This more-locked installation should probably be tied in with wheels (re: #558). It may even directly involve using pip to keep things explicit (kinda relates to #1012), but just have more known-good ways to handle in the poetry commands or flags.

two related/compatible use-cases

(a) poetry -> reqs, per @drunkwcodes example, and helps address #697

poetry export --ignore-conflicts
# [as needed] "Resolve conflicts in requirements.txt" - manually, or interactive prompt option? 
pip install -r requirements.txt

(b) reqs -> wheels -> install-with-pip

... installing with pip doesn't necessarily conflict with Poetry, you can do the pip-install-from-wheels / with-whatever-options, then call poetry install and it will do almost nothing if things look good, otherwise it'll install just the little bit of drift.

... seems to help address #558, and by directly exposing pip may make sense for cases like #1012

poetry export --ignore-conflicts 
# [as needed] "Resolve conflicts in requirements.txt"  - manually, or interactive prompt option? 
pip wheel -r requirements.txt  # assumption: you've set PIP_WHEEL_DIR
pip install --find-links "file://$PIP_WHEEL_DIR" -r requirements.txt

imagining new options to bring things together

implicit in example below: poetry export should take an argument for what its output is named (currently it is hardcoded to requirements.txt, it should take a filename or allow stdout output (convention being -))

intermediate idea 1

pip wheel -r <( poetry export --ignore-conflicts --output - )  # pip respects PIP_WHEEL_DIR or --wheel-dir, see pip docs
poetry install --find-links="file://$PIP_WHEEL_DIR"  # ..? does it need reqs again?

intermediate idea 2 - seems clean to me, but could be disagreeable to some

poetry export --ignore-conflicts --fetch-wheels --wheel-output=./wheelhouse --output=poetry.wheelhouse.lock
poetry install --find-links=./wheelhouse --extra-lock=./poetry.wheelhouse.lock

the idea in that second one being, in the event of a conflict, poetry would fall back to the --extra-lock file that was given. Or maybe call it --extra-constraints and .txt instead of .lock, if it seems like concepts are getting blurry.

intermediate idea 3 - does require one pip call, but very explicit?

If that's too weird, this is a fairly clean compromise:

poetry export --ignore-conflicts --wheel-output=./wheelhouse --output=poetry.wheelhouse.txt
pip install --find-links "file://./wheelhouse" -r poetry.wheelhouse.txt
poetry install --find-links=./wheelhouse

... in that last idea, poetry install is not even paying attention to poetry.wheelhouse.txt (which is just a requirements.txt, just trying to name it so its origin and usage context are clear). it's using its own pyproject.toml/poetry.lock, only, in that one; but it would prefer the wheelhouse before using PyPI.

poetry install would just happen to ignore already-installed cases and not hit conflicts, based on that pip install step.

(I also omitted --fetch-wheels because maybe that can just be implicit from the --wheel-output flag being present.)

interactive conflict resolution?

In those three examples I was trying to suggest ways that the intermediate manual step of hand-resolving conflicts, could be skipped, yet still having determinate results (if one checks the export'd artifact into VCS.

There could still be cases where something like,

poetry export --interactive-conflicts

and/or

poetry lock --interactive-conflicts

could be relevant; with prompts asking what choice is preferred. Since they are supervised maintenance tasks they could be interactive ...

p.s. pip flags in general

Also there still seems to be a general need to allow most/all pip options/env-vars/flags to be controlled from Poetry config and/or CLI. Will help a number of use cases.

[drunkwcodes]

I think the command can even be renamed to poetry resolve, which only outputs conflict warnings or errors by default, and has -o, -v, --tree options. Since somtimes we need the dependency resolution result but not to export.

• -o option is same as poetry export -f
• -v option means verbose and shows debug messages of the resolver.
• --tree option will lists dependency graphics in terminal.

P.S Reply to interactive mode:

interactive conflict resolution?

In those three examples I was trying to suggest ways that the intermediate manual step of hand-resolving conflicts, could be skipped, yet still having determinate results (if one checks the export'd artifact into VCS.

Spawn a process to open text editor like vi[m], emacs, nano etc. right after file generation sounds adequate.

[floer32]

That sounds really great. Less steps the better and your suggestion coheres very well with the existing functionality and the intentions we are discussing.

[epage]

So the use cases we've been running into with proper dependency resolution: packages we get value out of that have low activity or are unmaintained that happen to work with newer versions of dependencies. This is much worse than in Rust because Rust supports multiple versions of a dependency existing in your dependency graph. Here, we need one compatible version through the entire graph.

The proposed solution for this seems to rely on manually resolving the conflcit on every upgrade. This seems tedious for a process (poetry update) that we should be able to automate (i.e. via Dependabot). Personally, my preference would be for the override to be recorded in the pyproject.toml file so it can be a persistent input to the process. For example, we could have a table of patches to the dependency graph that override the version constraint.

[danields761]

Strict dependency resolution reveals a lot of issues with a lot of packages in Python ecosystem. There should be a least a way to handle dependencies by yourself without the need to contact with package maintainer, wait for patches etc.

I am agree with @epage that there should be a way to override dependency-graph

[kylebebak]

This makes poetry very difficult to use in all sorts of real-world projects.

This problem exists with other package managers, and has been solved before. yarn, for example, handles a similar (but more complicated) problem with resolutions: https://yarnpkg.com/lang/en/docs/selective-version-resolutions/.

It's also worth noting that pip will let you install deps that are incompatible with other deps, and will print something like the following to the console:

django-stubs 0.12.1 has requirement mypy<0.710,>=0.700, but you'll have mypy 0.711 which is incompatible.
Installing collected packages: mypy
Successfully installed mypy-0.711

I totally agree that poetry's default behavior should be to prevent users from installing packages whose deps can't be properly resolved, but users need to be able to opt out of this restriction (as they can with pip), because it's a common real-world situation. Telling users that their best option is to open issues with every package that has stale or incorrect version requirements is not realistic.

My vote is to keep the implementation as simple as possible, and to make the user responsible for managing the risk of ignoring conflicts. After all, it would be opt-in behavior.

I propose making this a flag on poetry add: poetry add <package> --ignore-conflicts. This would add an ignores-conflicts flag to the dependency in pyproject.toml, in much the same way --allow-prereleases adds an allows-prereleases flag.

[zehauser]

Unfortunately this continues to be an enormous issue that prevents using poetry in many real-world projects. 😢Which sucks, because poetry is awesome and getting awesomer!

Of course there won't be any solution overnight, but it would be great to get an updated opinion on this subject from @sdispater or another maintainer/triager (@stephsamson?).

[kylebebak]

@sdispater @stephsamson

Any update on this? I think this thread makes it clear that being able to ignore conflicts on an opt-in basis is necessary to make Poetry usable as a dependency manager for a wide range of projects.

I just want to reiterate that pip does it. It sensibly warns the user that there are version conflicts, but it doesn't prevent them from installing dependencies. There's no reason Poetry should prevent users from doing this either.

It's obviously outside the purview of Poetry to make sure that the dependency requirements of every Python package ever made are defined correctly.

How difficult would it be to implement this as proposed above, so that it works like --allow-prereleases?

[sdispater]

@kylebebak pip does it as a side-effect of not having a proper dependency resolver and, as far as I know, it's not a feature.

I understand the points made here but like I said before this is not a trivial change and would require a lot of work on the dependency resolution front that I am not willing to do. If someone want to step in and implement it in an intuitive manner I will gladly take a look at it.

But this leaves a lot of uncertainties, let's take an example:

• A depends on C >= 1.0.0, <2.0.0
• B depends on C >= 2.0.0, <3.0.0
• C intoduced API-breaking changes in version 2.0.0
• You force C to the range >= 2.0.0, <3.0.0

Now, you might introduce bugs in the behavior of A because of the API change while using a lower version of B might be the solution to the issue. The resolver and its conflict reporting is here to make you think about your dependencies and understand what is going on. Dependency constraints are here for a reason and introducing a way to override them introduces a risk of it being misused instead of finding the proper solution.

That being said, if someone find a good solution to this problem and implement it, I'd gladly review it.

[kylebebak]

@sdispater

When he created this issue, @zehauser mentioned a reason that provides a very good justification for this feature:

But... many packages out there are not specifying their dependencies properly. Even if they are, there's always the possibility that their specified dependencies are a tighter range than they strictly need to be.

Imagine you depend on A and B. Let's say A depends on B >= 1.1.0, <1.3.0, but it turns out A works just fine if you use B v1.9 instead of B v1.1.

Because A has specified an excessively narrow version range for B, you now can't ever upgrade B's version if you use Poetry, even though newer versions of B work just fine with A.

Your only option is to ask A's maintainer to broaden the version range. Maybe he'll do this in a week, maybe he'll do it in a month, and maybe he'll do it never...

[brycedrennan]

This has also prevented me from using poetry for projects.

I think part of the tension here is between the differing needs of applications vs libraries. For libraries, allowing conflicting dependencies would be bad. For applications, its sometimes necessary to use libraries that claim to be conflicting because you have to get stuff done and cant wait on a maintainer to update their library.

Perhaps any solution we come up with should allow overriding conflicts for features used by apps (poetry install) but not for building packages (poetry build)?

[nikordaris]

I think the bottom line is if your application doesn't directly use library C but A and B use conflicting versions then poetry should fail the install. But if through testing you discover that the limited usage of C by A and B turns out to only use portions that were non breaking changes then you should be allowed to set the version of C that makes sense in your application. The expectation of a dep manager isn't an enforcement tool, it's a convenience tool for efficiently installing all my indirect dependencies such that it won't cause conflicts. Poetry does a great job of this. The only thing missing is the ability for the dev to say I know better for this edge case so actually install this version instead, I'll assume all risk in my decision.

With that said, it sounds like this is a hard change so can we start talking about ways we could solve this now instead of whether we should?

[fakyras]

This also would help to resolve issues like https://github.com/sdispater/poetry/issues/1330 by manually adding functools32 to ignored package list - a little bit hacky, but a valid solution.

I have been using poetry for over a year now, but this situation is very disappointing. It is practically impossible to use poetry together with tensorflow2 at current state :(

[windviki]

This also would help to resolve issues like #1330 by manually adding functools32 to ignored package list - a little bit hacky, but a valid solution.

I have been using poetry for over a year now, but this situation is very disappointing. It is practically impossible to use poetry together with tensorflow2 at current state :(

+1 By now tensorflow2 has functools32 dependency issue and it prevents me to use poetry.

[dmontagu]

@windviki you can just manually delete functools32 from the poetry.lock and everything works. At least, that works for me.

[windviki]

@windviki you can just manually delete functools32 from the poetry.lock and everything works. At least, that works for me.

Thanks

[zehauser]

@dmontagu's workaround is good to know, but not going to help me (and I suspect many other people) use Poetry in these situations. For me at least, manually modifying poetry.lock is not going to fly for anything other than solo projects.

[rendaw]

Modifying poetry.lock didn't work for me, poetry install then complains and refuses to install anything.

I probably shouldn't be sticking my neck in here, but say you want to remove or use a manually specified version of X that one of dependents A and B claim to be incompatible with. Isn't it just a matter of ignoring links from A->X and B->X (or more generally *->X) when building constraints for X, and just using the unary constraint list of what was manually specified?

Or anything really here - even a command line override when installing that leaves the files alone might be able to get us back up in prod here.

I urge you to reconsider supporting this feature. To me Poetry is the alternative to Pipenv that respects the real world needs of its users. Saying "all your dependencies will always perfectly resolve, so you don't need manual overrides" feels similar to saying "minor version upgrades are always backwards compatible, so you can't complain if we upgrade all your dependencies at our whim" which is the reason I dropped Pipenv and tried Poetry in the first place.

[dmontagu]

@rendaw Are you sure you removed precisely the right references? There are a few places you have to remove things. What was the error from poetry install?

@zehauser If you are version controlling the poetry.lock (which you should be), I don't understand why manually editing the lock file is unacceptable on non-solo projects. Yeah, it's a little more annoying than having it fully automatically generated, but plenty of large-scale projects get by just fine with manual management of a requirements.txt file; this seems easily superior to that alternative.

With a good git diff viewer, it seems plenty easy to maintain the lock file, even when you want to update the dependency that causes the problems -- if you ever need to generate the lock file from scratch you can just review the diffs and make edits as necessary. (That's actually what I have been doing in my tensorflow projects without a problem.)

That said, I wouldn't mind if there was an easier way to override other projects' dependency mistakes.

[rendaw]

I don't understand why manually editing the lock file is unacceptable on non-solo projects

I think the risk here is that any command that modifies the lock file could potentially lead to the deleted dependency being restored. If a dev wasn't aware/forgot that a dependency was explicitly tweaked they won't know to re-remove it, and there's already a lot of lockfile noise (hash updates, constraint changes, other dependencies adding/removing transitive dependencies, etc.) that there's no real way to review anyway and I tend to not look at closely.

@rendaw Are you sure you removed precisely the right references?

No :sweat_smile: . I did a search and removed the dep block itself, plus it's line in the hashes from the lock file. I tried removing the line referring to the dep from the dependent project in the lockfile too, and restored that when I got the above error, restored it and got the same error. Luckily the dep was only one level deep - I ended up forking our primary dep and pointed poetry at the branch. But that's caused us pain before, when we forgot we had a project depending on the branch and removed it.

[dmontagu]

Yeah, I agree there is a lot of noise and ugliness to deal with, but that's why it's a work around and not a first class solution. I'm still not sure what the alternative is.

I'm sure it would be easy to write a python script that performs the precise modifications you want to the lockfile, and just add it to your Makefile / CI / etc. While unfortunate that it's necessary, it would probably still be substantially less work than porting to a different (and likely inferior in most ways) package management solution.

[frnhr]

Yet another approach could be a new file, .working_versions (or section in pyproject.toml) where we could manually specify that some package actually works with some version of a dependency.

So taking the example from @kylebebak above https://github.com/sdispater/poetry/issues/697#issuecomment-524118181

Imagine you depend on A and B. Let's say A depends on B >= 1.1.0, <1.3.0, but it turns out A works just fine if you use B v1.9 instead of B v1.1.

We could add this to .working_versions file:

[A]
B = ["1.9"]

This translates to "A works fine with B=1.9".

I haven't taken a look at the internals of poetry, but it seems like this would be a pretty non-invasive approach. @sdispater is there any merit in this?

When doing poetry install, poetry could (should?) complain with a warning that this is an unsupported dependency, risky, and kittens might die.

Likewise, when the upstream package (namely, "A") updates its dependencies, poetry can maybe notify that we have an unnecessary entry in ".working_versions" file.

[nikordaris]

I have not looked at the code for poetry but I don't understand why the resolution order can't just take the version specified in a direct dep over indirect ones. Example:

py-a/requirements.txt py-c==2.0

py-b/requirements.txt py-c>=1.0,<2.0

I've determined that version py-c 2.0 actually works with how I use py-b in my app. The version range is not incorrect in py-b and therefore should not be updated but the limited usage of py-b in my app doesn't actually execute the breaking changes made in py-c 2.0. Therefore I want to tell poetry it is ok to use py-c 2.0 in my app. I will then make a comment in my code somewhere documenting the risk with further usage of py-b. So i configure my pyproject.toml to the following to signal to poetry that i explicitly want 2.0 version for py-c library.

my-app/pyproject.toml

[tool.poetry.dependencies]
py-a = "~1.0"
py-b = "~2.0"
py-c = "2.0"

[apryiomka]

I do agree the fact that poetry doesn't allow to set ranges / overwrite for transitive dependencies does seem like a BIG issue for real time applications and adoption. The argument like

I don't want Poetry to have to make up for a lack of proper tools or proper specifications, there is already a lot of work to be done as it is.

is exactly why poetry is not our tool of choice for the enterprise application and we have to use Pipfile. Like many others stated, we do use 3rd party packages and we cannot guarantee the packages metadata as we don't own them. it does't though diminish the utility of the package, but we do need to add some overwrites and not purely rely on poetry's resolution.

What we really need is a section in pyproject.toml that would allow us to overwrite any package installation without requiring the package in the dependencies.

[apryiomka]

I think the risk here is that any command that modifies the lock file could potentially lead to the deleted dependency being restored. If a dev wasn't aware/forgot that a dependency was explicitly tweaked they won't know to re-remove it, and there's already a lot of lockfile noise (hash updates, constraint changes, other dependencies adding/removing transitive dependencies, etc.) that there's no real way to review anyway and I tend to not look at closely.

I actually see little to no value in making the lock file so complex, including the metadata information and SHAs. A list with flattened versions should be more than sufficient in my opinion.

[nikordaris]

@apryiomka the point is the lock file is generated and so changes to it run the risk of getting blown away the next time it gets generated. Also, the SHA is a good security practice for ensuring the package hasn't been modified. I don't think humans are really the intended audience of lock files.

[dmontagu]

I actually see little to no value in making the lock file so complex, including the metadata information and SHAs. A list with flattened versions should be more than sufficient in my opinion.

I think checking SHAs is a critical component of ensuring the security of a software supply chain.

I do wish there was better support for overriding dependency versions precisely for the reasons described by others in this issue, but I don't think that should come at the expense of checking SHAs (one of the most important parts of a lockfile used with a package manager for any language, in my opinion).

[adiled]

That's quite some neurons firing there, can anyone @zehauser @swarmer @sdispater @danielkza @fsworld009 chime in on this naivette of mine. How feasible is it comparitively to have a hint regarding failing root dependency to the version that won't fail. That should be inline with the philosophy.

[seansfkelley]

Yarn has a great list of justifications for why you would want this. The way it implements it is pretty straightforward, too: a separate configuration block that just clobbers any version ranges declared by dependencies or sub-dependencies. This should dovetail nicely with normal dependency resolution, because it's just overriding existing version specifications with a different one, globally. As noted earlier in this thread, this is for applications, not libraries.

Here's the list from the link above, copied for posterity and lazy:

Why would you want to do this?

• You may be depending on a package that is not updated frequently, which depends on another package that got an important upgrade. In this case, if the version range specified by your direct dependency does not cover the new sub-dependency version, you are stuck waiting for the author.
• A sub-dependency of your project got an important security update and you don’t want to wait for your direct-dependency to issue a minimum version update.
• You are relying on an unmaintained but working package and one of its dependencies got upgraded. You know the upgrade would not break things and you also don’t want to fork the package you are relying on, just to update a minor dependency.
• Your dependency defines a broad version range and your sub-dependency just got a problematic update so you want to pin it to an earlier version.

The first bullet hits the nail on the head for my use-case. I'm waiting for a release of one of my dependencies that includes a trivial version bump of one of it's dependencies for 3.8 compatibility. Right now, my hands are tied, because the latest published version of this package is dragging in a 3.6-only sub-dependency. That means I have to either roll my project back to 3.6, fork the library I'm waiting for just to package the code it already has (!) on master, or hope that I don't need to make many changes to my project while I wait for the next release of this dependency at some indeterminate future point.

[wolever]

Hey folks! First off: thank you so much for Poetry! This is fantastic software.

Second: from reading through the discussion here, it seems like there are three reasonable resolutions to this problem:

1. A project-wide overrides which would globally override any package version requirements. For example, they might look something like:

$ cat pyproject.toml
...
[tool.poetry.overrides (THIS IS AN EXAMPLE)]
requests = "^2.18"
...
$ poetry add pyrebase
...
WARNING: pyrebase requires requests = 2.11.1 which is incompatible with project-wide override of requests = ^2.18
...

This has the advantage of being simple, easy to implement, and easy to issue warnings when there are potential incompatibilities (like in the example above). It could also be combined with some per-requirement metadata to suppress warnings when a particular version incompatibility is known to be acceptable. For example:

[tool.poetry.dependencies (THIS IS AN EXAMPLE)] 
pyrebase = { version = "3.0.27", ignore-resolution-warnings = { requests = "2.11.1" } }

2. Per-package version overrides:

$ poetry add --override requests=^2.18 pyrebase
WARNING: pyrebase requires requests = 2.11.1 which is incompatible with override of requests = ^2.18 (ignoring)

This has the advantage of being straightforward to implement (I've build a proof-of-concept), but has the disadvantages of being less user-friendly than option (3), and has the same problems with version changes as option (1).

3. A per-package --ignore-conflicts argument, which suppresses but tracks all version version conflicts:

$ poetry add --ignore-conficts pyrebase
WARNING: pyrebase requires requests = 2.11.1 which is incompatible current requirement of requests = ^2.18 (ignoring)
…
$ cat pyproject.toml
[tool.poetry.dependencies (THIS IS AN EXAMPLE)] 
pyrebase = { version = "3.0.27", conflict-resolutions = { requests = ["2.11.1", "^2.18"] } }

Where the semantics of conflict-resolutions would be "in the given package's dependency tree, replace all instances of the conflicting version (2.11.1, in this example) with the resolution version (^2.18)".

This ensures that:

1. Changes in the package's dependencies will be noticed (ex, if version pyrebase = 4.0 depends on requests = 2.12.0, then a dependency resolution error will be raised)
2. Changes in the project's dependencies will be noticed (ex, if the project dependencies change to requests = ^2.20', a dependency resolution error will be raised).

This has the advantage of being both safe and user-friendly, although likely more difficult to implement than either options (1) or (2).

In my (likely incomplete and not-entirely-informed) opinion, though, this seems like the best option.

If the maintainers would consider a PR for any of the above options, I would be happy to rough something out 🙂

[alicederyn]

We have a similar issue where there are two versions of a library, psycopg2 and psycopg2-binary. They differ only in that the latter is pre-built while the former is built on installation. Unfortunately, we need to use the former while our upstream libraries (django) have linked to the latter. This is not something we can ask them to change, because most users will want the simpler binary package.

[logangrado]

Any progress on this? This issue is preventing me from adopting poetry wholesale, I've encountered numerous issues of dependencies over-specifying their version constraints, making it impossible to use.

If there's a single solution the community has settled on, I'd be happy to take a pass at implementing it.

From reading through this thread, @wolever's solution seems like the best solution, specifically the --ignore-conflicts flag to poetry add

A per-package --ignore-conflicts argument, which suppresses but tracks all version version conflicts: $ poetry add --ignore-conficts pyrebase WARNING: pyrebase requires requests = 2.11.1 which is incompatible current requirement of requests = ^2.18 (ignoring) … $ cat pyproject.toml [tool.poetry.dependencies (THIS IS AN EXAMPLE)] pyrebase = { version = "3.0.27", conflict-resolutions = { requests = ["2.11.1", "^2.18"] } } Where the semantics of conflict-resolutions would be "in the given package's dependency tree, replace all instances of the conflicting version (2.11.1, in this example) with the resolution version (^2.18)".

[adiled]

@logangrado I have skimmed through this earlier and my interpretation was that maintainer(s) have a "change-maker" ideology, that's one reason poetry is so poetic. In my 10 years of experience benefiting from open-source, this cannot change without a split, however python community is largely built on scripting and isolated tooling than building monoliths, hence less people facing dependency problems, less diversity.

But we should at least have hinting on what versions would be compatible as I mentioned earlier, it's really a waste of time to manually figure it out.

[logangrado]

But we should at least have hinting on what versions would be compatible as I mentioned earlier, it's really a waste of time to manually figure it out.

@m-adilshaikh, what happens when there is not a compatible version available? For example, if you depend on foo and bar, both of which depend on baz, but ALL versions of foo depend on baz < x, and ALL versions of bar depend on baz >=x, then there is no compatible version which can be found.

I'm experiencing this with packages which specify dependencies exactly (e.g. someone just froze their venv and set that as the requirements ....). I need the ability to ignore their requirements.

[adiled]

@logangrado https://github.com/python-poetry/poetry/issues/697#issuecomment-524073420 Basing my judgements from that. In case there is not compatible version, it should state in bold, in-fact bold and red (or figure out some combinatorics that would work, however expensive that is). For example, I had to downgrade jupyter because of this and nudge a few utility packages but that task alone was bruteforce because the warning just gave a beautiful recursive tree output but I happen to be human.

I understand there are complex cases, edge cases that the above thread discusses with not much consensus but meanwhile diagnostic errors would be a progressive step.

[alicederyn]

There may be real-world cases where resolution is actually impossible, but there are also cases (like psycopg2 vs psycopg2-binary) where it can, and we just need the tool to let us do it.

[nikordaris]

1. A project-wide overrides which would globally override any package version requirements. For example, they might look something like:

$ cat pyproject.toml
...
[tool.poetry.overrides (THIS IS AN EXAMPLE)]
requests = "^2.18"
...
$ poetry add pyrebase
...
WARNING: pyrebase requires requests = 2.11.1 which is incompatible with project-wide override of requests = ^2.18
...

2. Per-package version overrides:

$ poetry add --override requests=^2.18 pyrebase
WARNING: pyrebase requires requests = 2.11.1 which is incompatible with override of requests = ^2.18 (ignoring)

3. A per-package `--ignore-conflicts` argument, which suppresses but tracks all version version conflicts:

$ poetry add --ignore-conficts pyrebase
WARNING: pyrebase requires requests = 2.11.1 which is incompatible current requirement of requests = ^2.18 (ignoring)
…
$ cat pyproject.toml
[tool.poetry.dependencies (THIS IS AN EXAMPLE)] 
pyrebase = { version = "3.0.27", conflict-resolutions = { requests = ["2.11.1", "^2.18"] } }

Of @wolever's options, I prefer Option 1. Doing per requirement overrides is cumbersome and introduces another layer of conflict resolution between all of your requirement's overrides. How do you resolve conflicting overrides between multiple requirements? IMO, the cleanest solution is to authoritatively declare the specific version range that is compatible with your application. Option 1 does this by defining a single list of requirements that should trump all other derived requirements. I personally would take it a step further and just say any requirements in [tool.poetry.dependencies] override any child dependency. This can be described as Option 4:

4. Derived dependencies fallback to explicitly set dependencies if version resolution is unsuccessful.

   $ cat pyproject.toml
   ...
   [tool.poetry.dependencies (THIS IS AN EXAMPLE)]
   requests = "^2.18"
   ...
   $ poetry add pyrebase
   ...
   ERROR: pyrebase requires requests = 2.11.1 which is incompatible with requests = ^2.18
   ...
   $ poetry add --ignore-conficts pyrebase
   ...
   WARNING: pyrebase requires requests = 2.11.1 which is incompatible with requests = ^2.18
   ...
   $ cat pyproject.toml
   ...
   [tool.poetry.dependencies (THIS IS AN EXAMPLE)]
   requests = "^2.18"
   pyrebase = "3.0.27"
   ...
   $ poetry install
   ...
   - Installing requests (2.18)
   - Installing pyrebase (3.0.27)
   ...
   WARNING: pyrebase requires requests = 2.11.1 but you will have requests = 2.18

[djerraballi]

Having a command that woudl explicitly drop conflicts to "Warnings" instead of failing the install would be great for a lot fo reasons.

Unfortunately in the python ecosystem, there are a lot of great libraries with terrible terrible setup.py configurations. Versions requirements that are far too strict, that impede development.

I think it makes the most sense that when a package is explicitly present with a version in pyproject.toml, that with the proper option, that package would be pulled in at that version regardless of subdependency setup.py requirements.

[andrewleech]

I'd like to add my vote for allowing explicit package versions listed in pyproject.toml to override any library dependencies implicit version, with a warning printed that this is happening.

I'm having a similar issue to others here, I've been working on a new library that uses aws. I've hit aiobotocore depends on botocore (>=1.17.44,<1.17.45) but my library required botocore (^1.19.7) for authentication to work (didn't work on older versions). In practice aiobotocore seems to be working fine with the newer botocore and I can ping them to try to get their dependency list updated, but in the mean time I've had to remove my usage of poetry from the library so I can release something that can be installed.

[muppetjones]

Another reason for sub-dependency overrides: typing. The backport is necessary for python 3.4 and earlier, and some older packages may have typing as a dependency. Python 3.5-7 will ignore the dependency (as they have their own); however, v3.8 will not ignore it, which has a tendency to break things:

    File "<project>/.tox/py38/lib/python3.8/site-packages/typing.py", line 1007, in __new__
      self._abc_registry = extra._abc_registry
  AttributeError: type object 'Callable' has no attribute '_abc_registry'

In my pyproject.toml, I do have

typing = { version = "*", python = "<=3.4" }

Unfortunately, poetry (and pip) install it anyway as a separate dependency installs it.

Another example is the awscli, which uses an older library with a known vulnerability...but they've said they will not update their requirements b/c their package doesn't use the code that is vulnerable. (This was a bit ago and may have been resolved?) The awscli package worked fine with the updated version but was range pinned to the vulnerable version. I can't recall exactly which package it was -- a (sub-)dependency of requests, I think.

[philmade]

Poetry is great, but I now have to uninstall it and use pip again because of this issue.

[finswimmer]

Hello everyone,

please let me cite @sdispater again:

I don't think this is desirable. This would require a lot of work and add complexity to the resolver - which is already complex due to the Python specific ecosystem - because of some packages specifying their dependencies poorly. I don't want Poetry to have to make up for a lack of proper tools or proper specifications, there is already a lot of work to be done as it is.

This is the job of each package's maintainers to ensure their dependencies are correct and loose enough to not create conflict.

If we ever want to have an ecosystem similar to what other languages already have, we have to draw the line somewhere and enforce everyone to contribute to the common goal. Poetry helps with that by making it easier to build and manages Python projects.

This is why I will close the issue now.

I haven't tried pip's new dependency resolver. But if they made it right, everyone will run into the same "problem" with it. And that's good for the python ecosystem, as it forces more and more package developer to proper specify their needs.

fin swimmer