search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.37k stars 2.26k forks source link

pathological version solving? (django-distill + boto3) #8823

Open anentropic opened 9 months ago

anentropic commented 9 months ago

[tool.poetry.dependencies] python = "^3.11"

[build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"


<!-- All the below steps should be completed before submitting your issue. Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) stable Poetry version, installed using a [recommended method](https://python-poetry.org/docs/#installation).
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] I have consulted the [FAQ](https://python-poetry.org/docs/faq/) and [blog](https://python-poetry.org/blog/) for any relevant entries or release notes.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option) and have included the output below.

## Issue
I can reproduce this in an empty poetry project.

1. attempt to `poetry add -vv 'django-distill[amazon]'`

output is like:

Resolving dependencies... 1: fact: poetrytemp is 0.1.0 1: derived: poetrytemp 1: fact: poetrytemp depends on django-distill (^3.1.3) 1: selecting poetrytemp (0.1.0) 1: derived: django-distill[amazon] (>=3.1.3,<4.0.0) 1: fact: django-distill (3.1.3) depends on django-distill (3.1.3) 1: fact: django-distill (3.1.3) depends on django () 1: fact: django-distill (3.1.3) depends on requests () 1: fact: django-distill (3.1.3) depends on boto3 () 1: selecting django-distill[amazon] (3.1.3) 1: derived: boto3 1: derived: requests 1: derived: django 1: derived: django-distill (==3.1.3) 1: fact: django-distill (3.1.3) depends on django () 1: fact: django-distill (3.1.3) depends on requests () 1: selecting django-distill (3.1.3) 1: fact: django (5.0) depends on asgiref (>=3.7.0) 1: fact: django (5.0) depends on sqlparse (>=0.3.1) 1: fact: django (5.0) depends on tzdata () 1: selecting django (5.0) 1: derived: tzdata 1: derived: sqlparse (>=0.3.1) 1: derived: asgiref (>=3.7.0) 1: fact: requests (2.31.0) depends on charset-normalizer (>=2,<4) 1: fact: requests (2.31.0) depends on idna (>=2.5,<4) 1: fact: requests (2.31.0) depends on urllib3 (>=1.21.1,<3) 1: fact: requests (2.31.0) depends on certifi (>=2017.4.17) 1: selecting requests (2.31.0) 1: derived: certifi (>=2017.4.17) 1: derived: urllib3 (>=1.21.1,<3) 1: derived: idna (>=2.5,<4) 1: derived: charset-normalizer (>=2,<4) 1: selecting urllib3 (2.1.0) 1: selecting certifi (2023.11.17) 1: selecting charset-normalizer (3.3.2) 1: selecting idna (3.6) 1: selecting sqlparse (0.4.4) 1: selecting asgiref (3.7.2) 1: fact: boto3 (1.34.7) depends on botocore (>=1.34.7,<1.35.0) 1: fact: boto3 (1.34.7) depends on jmespath (>=0.7.1,<2.0.0) 1: fact: boto3 (1.34.7) depends on s3transfer (>=0.10.0,<0.11.0) 1: selecting boto3 (1.34.7) 1: derived: s3transfer (>=0.10.0,<0.11.0) 1: derived: jmespath (>=0.7.1,<2.0.0) 1: derived: botocore (>=1.34.7,<1.35.0) 1: fact: s3transfer (0.10.0) depends on botocore (>=1.33.2,<2.0a.0) 1: selecting s3transfer (0.10.0) 1: fact: botocore (1.34.7) depends on jmespath (>=0.7.1,<2.0.0) 1: fact: botocore (1.34.7) depends on python-dateutil (>=2.1,<3.0.0) 1: fact: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1) 1: derived: not botocore (==1.34.7) 1: fact: no versions of botocore match >1.34.7,<1.35.0 1: conflict: no versions of botocore match >1.34.7,<1.35.0 1: derived: not botocore (>1.34.7,<1.35.0) 1: conflict: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1) 1: ! botocore (==1.34.7) is partially satisfied by not botocore (>1.34.7,<1.35.0) 1: ! which is caused by "no versions of botocore match >1.34.7,<1.35.0" 1: ! thus: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1) 1: fact: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1) 1: derived: not botocore (>=1.34.7,<1.35.0) 1: derived: not boto3 (==1.34.7) 2: selecting certifi (2023.11.17) 2: selecting charset-normalizer (3.3.2) 2: selecting idna (3.6) 2: selecting sqlparse (0.4.4) 2: selecting asgiref (3.7.2) 2: fact: boto3 (1.34.6) depends on botocore (>=1.34.6,<1.35.0) 2: fact: boto3 (1.34.6) depends on jmespath (>=0.7.1,<2.0.0) 2: fact: boto3 (1.34.6) depends on s3transfer (>=0.10.0,<0.11.0) 2: selecting boto3 (1.34.6) 2: derived: s3transfer (>=0.10.0,<0.11.0) 2: derived: jmespath (>=0.7.1,<2.0.0) 2: derived: botocore (>=1.34.6,<1.35.0) 2: fact: s3transfer (0.10.0) depends on botocore (>=1.33.2,<2.0a.0) 2: selecting s3transfer (0.10.0) 2: fact: botocore (1.34.6) depends on jmespath (>=0.7.1,<2.0.0) 2: fact: botocore (1.34.6) depends on python-dateutil (>=2.1,<3.0.0) 2: fact: botocore (1.34.6) depends on urllib3 (>=1.25.4,<2.1) 2: derived: not botocore (==1.34.6) 2: fact: no versions of botocore match >1.34.6,<1.34.7 2: conflict: no versions of botocore match >1.34.6,<1.34.7 2: derived: not botocore (>1.34.6,<1.34.7) 2: conflict: botocore (1.34.6) depends on urllib3 (>=1.25.4,<2.1) 2: ! botocore (==1.34.6) is partially satisfied by not botocore (>1.34.6,<1.34.7) 2: ! which is caused by "no versions of botocore match >1.34.6,<1.34.7" 2: ! thus: botocore (>=1.34.6,<1.34.7) requires urllib3 (>=1.25.4,<2.1)

this continues forever in a fruitless search, downloading the wheel for every minor version of botocore

the problem seems to be something to do with `urllib3`

1: fact: requests (2.31.0) depends on urllib3 (>=1.21.1,<3)

1: derived: urllib3 (>=1.21.1,<3)

1: selecting urllib3 (2.1.0)

1: fact: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1)

1: conflict: botocore (1.34.7) depends on urllib3 (>=1.25.4,<2.1) 1: ! botocore (==1.34.7) is partially satisfied by not botocore (>1.34.7,<1.35.0) 1: ! which is caused by "no versions of botocore match >1.34.7,<1.35.0" 1: ! thus: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1) 1: fact: botocore (>=1.34.7,<1.35.0) requires urllib3 (>=1.25.4,<2.1)


it's not clear from the output why https://pypi.org/project/urllib3/1.26.18/ couldn't be used to satisfy `>=1.25.4,<2.1`

Ok but then the odd thing is if I do this:

1. `poetry add boto3`
2. `poetry add 'django-distill[amazon]'`

then it works fine!

If we look at the setup.py for django-distill: https://github.com/meeb/django-distill/blob/v3.1.3/setup.py#L30
it specifies:
```python
    extras_require = {
        'amazon': ['boto3'],

So no version constraint on boto3.

It doesn't make much sense that dependency resolution fails one way but not the other?

dimbleby commented 9 months ago

boto3 and urllib3 are known to be an unhappy pairing, you will find lots of similar issues if you search. Current view is that there is nothing much to be done about it, sorry.

please close.

d33bs commented 8 months ago

Thanks for opening this issue @anentropic ! I ran into a similar pattern recently and found that the following the workaround you mentioned helped too.

@dimbleby - thanks too for your highlighting the related elements and that there may be a pattern at hand! If it already exists, could you link to the relevant issue, PR, or discussion where boto3 and urllib3 incompatibilities are tracked? I'd mostly suggest this to be sure others who run into the same problem can understand the context and make sure a resolution eventually is addressed. Even if this isn't directly a Poetry issue to solve, it does exhibit itself through the use of the tool, so it could be helpful to make sure others are informed (especially as this can result in a lot of time through attempted dependency resolution).

dimbleby commented 8 months ago

@d33bs

you will find lots of similar issues if you search

please do that yourself if you want to find the similar issues, feel free to add links

Northo commented 8 months ago

Thanks for the workaround, @anentropic!

As @d33bs writes, this issue exhibits itself through Poetry, so some warning or other user facing information in these situations would be great!

ps. for future searches, we hit this when installing fiftyone

anentropic commented 8 months ago

similar issues:

So there are two fixes users could take right now:

FWIW pdm seems to cope with this fine:

$ pdm add 'django-distill[amazon]'
Adding packages to default dependencies: django-distill
🔒 Lock successful
Changes are written to pyproject.toml.
Synchronizing working set with resolved packages: 15 to add, 0 to update, 0 to remove

  ✔ Install asgiref 3.7.2 successful
  ✔ Install idna 3.6 successful
  ✔ Install jmespath 1.0.1 successful
  ✔ Install python-dateutil 2.8.2 successful
  ✔ Install requests 2.31.0 successful
  ✔ Install charset-normalizer 3.3.2 successful
  ✔ Install s3transfer 0.10.0 successful
  ✔ Install six 1.16.0 successful
  ✔ Install sqlparse 0.4.4 successful
  ✔ Install urllib3 2.0.7 successful
  ✔ Install django-distill 3.1.3 successful
  ✔ Install django 5.0.1 successful
  ✔ Install boto3 1.34.35 successful
  ✔ Install certifi 2024.2.2 successful
  ✔ Install botocore 1.34.35 successful

🎉 All complete!

all resolved in a reasonable amount of time.

So I wonder what they're doing differently.

dimbleby commented 8 months ago

So I wonder what they're doing differently.

it's just good or bad luck whether your solver happens first to explore a path that fixes boto3 first (when it is very easy to find a satisfactory urllib3) or a path that fixes urllib3 first (when it is very hard to find a satisfactory boto3)

pdm has its own pathological cases!

dimbleby commented 8 months ago

the most useful thing you can do right now, for future-you and the rest of the ecosystem, is to go and offer merge requests to django-distill or fiftyone or whoever, putting a (recent) lower bound on their boto3 dependency.

Then no installer is exposed to having to backtrack through the thousands of versions of boto3 that amazon release

MajorDallas commented 1 month ago

I've run into this issue with Celery 5.4.0. I don't need boto3 or the related sqs package that Celery directly has as an optional dependency, so the workaround I want is to simply exclude the sqs extra. I thought that having extras set in pyproject.toml would mean that those and only those extras would be pulled in, but despite setting it with exactly the extras I do want for Celery (and Kombu) it continues to insist on downloading every single release of boto3.

It usually takes about 950 seconds before it starts downloading anything. I'm a little afraid that setting Celery to 5.4.0 will make all poetry lock runs in the future take 20+ minutes because of a package I don't even want and am trying to exclude.

Edit: Apparently the pytest-celery package (from my dev group) can also pull in sqs and that was why Poetry seemed to be ignoring the extras I gave for Celery itself. I didn't find that sooner because boto3 was being installed for the first time and so poetry show --tree didn't list these packages and what was pulling them in. Overall an incredibly frustrating experience, but at least fixing that has it working in the usual 5 seconds or so.

dimbleby commented 1 month ago 
$ time poetry add --lock celery=5.4.0 pytest-celery
Using version ^1.1.1 for pytest-celery

Updating dependencies
Resolving dependencies... (3.7s)

Writing lock file

real    0m5.155s
user    0m2.697s
sys     0m0.148s

hard to guess what happened to you, but neither celery nor pytest-celery is problematic

MajorDallas commented 1 month ago

Probably a result of changing several dependency specs all at once, and one hack that I remember made sense at the time but was probably the real culprit.

[tool.poetry.dependencies]
-celery = {version="^5.3.6", extras=["redis", "tblib", "auth"]}
+celery = {version="5.4.0", extras=["redis", "tblib", "auth", "gevent"]}

 [tool.poetry.group.dev.dependencies]
-pytest = "^4.6"
# this is the hack: re-declaring celery in the dev group to add the pytest extra.
-celery = {version="*", extras=["pytest"]}
+pytest = "*"
# What I should have done in the first place was just declare pytest-celery directly:
+pytest-celery = {version="*", extras=["rabbitmq", "redis"]}
# With this, it's working like normal even with celery pinned to 5.4.0, the change that triggered this whole thing.

When I ran poetry update celery or poetry lock, it would take 20 minutes to get to the point where it started downloading every botocore release. I never let it run to completion, and the furthest it got after several attempts was botocore 1.15.44 (that particular run was 1098.3s).

It seems that having celery[pytest] was equivalent to having pytest-celery[all], which then pulled in sqs and thus boto3.

edit:

hrm, I thought the diff would make it pretty clear, but not really... The problematic config was:

[tool.poetry.dependencies]
celery = {version="5.4.0", extras=["redis", "tblib", "auth", "gevent"]}

[tool.poetry.group.dev.dependencies]
pytest = "*"
celery = {version="*", extras=["pytest"]}

when previously celery was set to 5.3.6 and pytest-celery matched to that automatically. I believe sqs is a new extra for both Celery and pytest-celery as of 5.4, but I could be wrong.

dimbleby commented 1 month ago

still locks in about five seconds for me.

Yes there are cases that cause a lot of backtracking. Quite possibly you hit one, though you are unable to say what it was.

But there still is nothing much to be done about it.

As I said back in February: the most useful thing you can do right now, for future-you and the rest of the ecosystem, is to go and offer merge requests to django-distill or fiftyone or whoever, putting a (recent) lower bound on their boto3 dependency.

Then no installer is exposed to having to backtrack through the thousands of versions of boto3 that amazon release

MajorDallas commented 1 month ago

Be that is it may, I probably could have resolved the case if I at least knew why botocore was being pulled in. I'd never even heard of botocore before yesterday, and poetry show was (correctly, I guess) saying there was nothing to show because botocore (and sqs and boto3) wasn't actually installed. It took most of the afternoon to figure out pytest-celery was the culprit. A simple message would have saved me a lot of time and frustration:

Updating dependencies
Resolving dependencies... 
pytest-celery[sqs] depends on botocore, which needs to be downloaded to continue dependency resolution.
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/ab/e5/4bf433f4fe4bb193f65c9d8be0b72d31c0ac04a564b787cc0f0929644325/botocore-1.17.26-py2.py
d33bs commented 1 month ago

Love the idea of a message which @MajorDallas suggested! Perhaps this isn't in scope for Poetry to fix, but developers often may have to dig deeply to find where the error may be coming from with little notification at the moment from Poetry by default. Maybe a timed notification for long-running procedures related to one dependency resolution could help! For example, if the resolution process is taking an excessive amount of time (i.e. longer than an hour or whatever is reasonable) an additional message appears during the process to notify the user about which dependency is causing the jam.

dimbleby commented 1 month ago

Verbose logging output already includes such information

MajorDallas commented 1 month ago

That is good to know, and I'm a little embarrassed it didn't occur to me try it :sweat_smile:

Still, I agree with d33bs: if it can be detected that things are taking longer than normal (say, more than 10 minutes or more than 30 backtracking steps), a message about what is being is resolved (without needing a v or three) would be nice. That makes it a simple decision for the user whether to wait it out or revise the declared dependencies.